Ethan A. Halm, MD, MPH

Hospital-acquired anemia (HAA) is defined as having a normal hemoglobin value upon admission but developing anemia during the course of hospitalization. The condition is common, with an incidence ranging from approximately 25% when defined by using the hemoglobin value prior to discharge to 74% when using the nadir hemoglobin value during hospitalization.^1-5 While there are many potential etiologies for HAA, given that iatrogenic blood loss from phlebotomy may lead to its development,^6,7 HAA has been postulated to be a hazard of hospitalization that is potentially preventable.⁸ However, it is unclear whether the development of HAA portends worse outcomes after hospital discharge.

The limited number of studies on the association between HAA and postdischarge outcomes has been restricted to patients hospitalized for acute myocardial infarction (AMI).^3,9,10 Among this subpopulation, HAA is independently associated with greater morbidity and mortality following hospital discharge.^3,9,10 In a more broadly representative population of hospitalized adults, Koch et al.² found that the development of HAA is associated with greater length of stay (LOS), hospital charges, and inpatient mortality. However, given that HAA was defined by the lowest hemoglobin level during hospitalization (and not necessarily the last value prior to discharge), it is unclear if the worse outcomes observed were the cause of the HAA, rather than its effect, since hospital LOS is a robust predictor for the development of HAA, as well as a major driver of hospital costs and a prognostic marker for inpatient mortality.^3,9 Furthermore, this study evaluated outcomes only during the index hospitalization, so it is unclear if patients who develop HAA have worse clinical outcomes after discharge.

Therefore, in this study, we used clinically granular electronic health record (EHR) data from a diverse cohort of consecutive medicine inpatients hospitalized for any reason at 1 of 6 hospitals to: 1) describe the epidemiology of HAA; 2) identify predictors of its development; and 3) examine its association with 30-day postdischarge adverse outcomes. We hypothesized that the development of HAA would be independently associated with 30-day readmission and mortality in a dose-dependent fashion, with increasing severity of HAA associated with worse outcomes.

Study Design, Population, and Data Sources

We conducted a retrospective observational cohort study using EHR data collected from November 1, 2009 to October 30, 2010 from 6 hospitals in the north Texas region. One site was a university-affiliated safety-net hospital; the remaining 5 community hospitals were a mix of teaching and nonteaching sites. All hospitals used the Epic EHR system (Epic Systems Corporation, Verona, Wisconsin). Details of this cohort have been published.^11,12This study included consecutive hospitalizations among adults age 18 years or older who were discharged from a medicine inpatient service with any diagnosis. We excluded hospitalizations by individuals who were anemic within the first 24 hours of admission (hematocrit less than 36% for women and less than 40% for men), were missing a hematocrit value within the first 24 hours of hospitalization or a repeat hematocrit value prior to discharge, had a hospitalization in the preceding 30 days (ie, index hospitalization was considered a readmission), died in the hospital, were transferred to another hospital, or left against medical advice. For individuals with multiple eligible hospitalizations during the study period, we included only the first hospitalization. We also excluded those discharged to hospice, given that this population of individuals may have intentionally desired less aggressive care.

Definition of Hospital-Acquired Anemia

HAA was defined as having a normal hematocrit value (36% or greater for women and 40% or greater for men) within the first 24 hours of admission and a hematocrit value at the time of hospital discharge lower than the World Health Organization’s sex-specific cut points.¹³ If there was more than 1 hematocrit value on the same day, we chose the lowest value. Based on prior studies, HAA was further categorized by severity as mild (hematocrit greater than 33% and less than 36% in women; and greater than 33% and less than 40% in men), moderate (hematocrit greater than 27% and 33% or less for all), or severe (hematocrit 27% or less for all).^2,14

Characteristics

We extracted information on sociodemographic characteristics, comorbidities, LOS, procedures, blood transfusions, and laboratory values from the EHR. Hospitalizations in the 12 months preceding the index hospitalization were ascertained from the EHR and from an all-payer regional hospitalization database that captures hospitalizations from 75 acute care hospitals within a 100-mile radius of Dallas-Fort Worth. International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) discharge diagnosis codes were categorized according to the Agency for Healthcare Research and Quality (AHRQ) Clinical Classifications Software (CCS).¹⁵ We defined a diagnosis for hemorrhage and coagulation, and hemorrhagic disorder as the presence of any ICD-9-CM code (primary or secondary) that mapped to the AHRQ CCS diagnoses 60 and 153, and 62, respectively. Procedures were categorized as minor diagnostic, minor therapeutic, major diagnostic, and major therapeutic using the AHRQ Healthcare Cost and Utilization Procedure Classes tool.¹⁶

Outcomes

The primary outcome was a composite of death or readmission within 30 days of hospital discharge. Hospital readmissions were ascertained at the index hospital and at any of 75 acute care hospitals in the region as described earlier. Death was ascertained from each of the hospitals’ EHR and administrative data and the Social Security Death Index. Individuals who had both outcomes (eg, a 30-day readmission and death) were considered to have only 1 occurrence of the composite primary outcome measure. Our secondary outcomes were death and readmission within 30 days of discharge, considered as separate outcomes.

Statistical Analysis

We used logistic regression models to evaluate predictors of HAA and to estimate the association of HAA on subsequent 30-day adverse outcomes after hospital discharge. All models accounted for clustering of patients by hospital. For the outcomes analyses, models were adjusted for potential confounders based on prior literature and our group’s expertise, which included age, sex, race/ethnicity, Charlson comorbidity index, prior hospitalizations, nonelective admission status, creatinine level on admission, blood urea nitrogen (BUN) to creatinine ratio of more than 20:1 on admission, LOS, receipt of a major diagnostic or therapeutic procedure during the index hospitalization, a discharge diagnosis for hemorrhage, and a discharge diagnosis for a coagulation or hemorrhagic disorder. For the mortality analyses, given the limited number of 30-day deaths after hospital discharge in our cohort, we collapsed moderate and severe HAA into a single category. In sensitivity analyses, we repeated the adjusted model, but excluded patients in our cohort who had received at least 1 blood transfusion during the index hospitalization (2.6%) given its potential for harm, and patients with a primary discharge diagnosis for AMI (3.1%).¹⁷

The functional forms of continuous variables were assessed using restricted cubic splines and locally weighted scatterplot smoothing techniques. All analyses were performed using STATA statistical software version 12.0 (StataCorp, College Station, Texas). The University of Texas Southwestern Medical Center institutional review board approved this study.

Of 53,995 consecutive medicine hospitalizations among adults age 18 years or older during our study period, 11,309 index hospitalizations were included in our study cohort (Supplemental Figure 1). The majority of patients excluded were because of having documented anemia within the first 24 hours of admission (n=24,950). With increasing severity of HAA, patients were older, more likely to be female, non-Hispanic white, electively admitted, have fewer comorbidities, less likely to be hospitalized in the past year, more likely to have had a major procedure, receive a blood transfusion, have a longer LOS, and have a primary or secondary discharge diagnosis for a hemorrhage or a coagulation or hemorrhagic disorder (Table 1).

Epidemiology of HAA

Among this cohort of patients without anemia on admission, the median hematocrit value on admission was 40.6 g/dL and on discharge was 38.9 g/dL. One-third of patients with normal hematocrit value at admission developed HAA, with 21.6% developing mild HAA, 10.1% developing moderate HAA, and 1.4% developing severe HAA. The median discharge hematocrit value was 36 g/dL (interquartile range [IQR]), 35-38 g/dL) for the group of patients who developed mild HAA, 31 g/dL (IQR, 30-32 g/dL) for moderate HAA, and 26 g/dL (IQR, 25-27 g/dL) for severe HAA (Supplemental Figure 2). Among the severe HAA group, 135 of the 159 patients (85%) had a major procedure (n=123, accounting for 219 unique major procedures), a diagnosis for hemorrhage (n=30), and/or a diagnosis for a coagulation or hemorrhagic disorder (n=23) during the index hospitalization. Of the 219 major procedures among patients with severe HAA, most were musculoskeletal (92 procedures), cardiovascular (61 procedures), or digestive system-related (41 procedures). The most common types of procedures were coronary artery bypass graft (36 procedures), hip replacement (25 procedures), knee replacement (17 procedures), and femur fracture reduction (15 procedures). The 10 most common principal discharge diagnoses of the index hospitalization by HAA group are shown in Supplemental Table 1. For the severe HAA group, the most common diagnosis was hip fracture (20.8%).

Predictors of HAA

Compared to no or mild HAA, female sex, elective admission status, serum creatinine on admission, BUN to creatinine ratio greater than 20 to 1, hospital LOS, and undergoing a major diagnostic or therapeutic procedure were predictors for the development of moderate or severe HAA (Table 2). The model explained 23% of the variance (McFadden’s pseudo R²).

Incidence of Postdischarge Outcomes by Severity of HAA

The severity of HAA was associated with a dose-dependent increase in the incidence of 30-day adverse outcomes, such that patients with increasing severity of HAA had greater 30-day composite, mortality, and readmission outcomes (P < 0.001; Figure). The 30-day postdischarge composite outcome was primarily driven by hospital readmissions given the low mortality rate in our cohort. Patients who did not develop HAA had an incidence of 9.7% for the composite outcome, whereas patients with severe HAA had an incidence of 16.4%. Among the 24 patients with severe HAA but who had not undergone a major procedure or had a discharge diagnosis for hemorrhage or for a coagulation or hemorrhagic disorder, only 3 (12.5%) had a composite postdischarge adverse outcome (2 readmissions and 1 death). The median time to readmission was similar between groups, but more patients with severe HAA had an early readmission within 7 days of hospital discharge than patients who did not develop HAA (6.9% vs. 2.9%, P = 0.001; Supplemental Table 2).

Association of HAA and Postdischarge Outcomes

In unadjusted analyses, compared to not developing HAA, mild, moderate, and severe HAA were associated with a 29%, 61%, and 81% increase in the odds for a composite outcome, respectively (Table 3). After adjustment for confounders, the effect size for HAA attenuated and was no longer statistically significant for mild and moderate HAA. However, severe HAA was significantly associated with a 39% increase in the odds for the composite outcome and a 41% increase in the odds for 30-day readmission (P = 0.008 and P = 0.02, respectively).

In sensitivity analyses, the exclusion of individuals who received at least 1 blood transfusion during the index hospitalization (n=298) and individuals who had a primary discharge diagnosis for AMI (n=353) did not substantively change the estimates of the association between severe HAA and postdischarge outcomes (Supplemental Tables 3 and 4). However, because of the fewer number of adverse events for each analysis, the confidence intervals were wider and the association of severe HAA and the composite outcome and readmission were no longer statistically significant in these subcohorts.

In this large and diverse sample of medical inpatients, we found that HAA occurs in one-third of adults with normal hematocrit value at admission, where 10.1% of the cohort developed moderately severe HAA and 1.4% developed severe HAA by the time of discharge. Length of stay and undergoing a major diagnostic or therapeutic procedure were the 2 strongest potentially modifiable predictors of developing moderate or severe HAA. Severe HAA was independently associated with a 39% increase in the odds of being readmitted or dying within 30 days after hospital discharge compared to not developing HAA. However, the associations between mild or moderate HAA with adverse outcomes were attenuated after adjusting for confounders and were no longer statistically significant.

To our knowledge, this is the first study on the postdischarge adverse outcomes of HAA among a diverse cohort of medical inpatients hospitalized for any reason. In a more restricted population, Salisbury et al.³ found that patients hospitalized for AMI who developed moderate to severe HAA (hemoglobin value at discharge of 11 g/dL or less) had greater 1-year mortality than those without HAA (8.4% vs. 2.6%, P < 0.001), and had an 82% increase in the hazard for mortality (95% confidence interval, hazard ratio 1.11-2.98). Others have similarly shown that HAA is common among patients hospitalized with AMI and is associated with greater mortality.^5,9,18 Our study extends upon this prior research by showing that severe HAA increases the risk for adverse outcomes for all adult inpatients, not only those hospitalized for AMI or among those receiving blood transfusions.

Despite the increased harm associated with severe HAA, it is unclear whether HAA is a preventable hazard of hospitalization, as suggested by others.^6,8 Most patients in our cohort who developed severe HAA underwent a major procedure, had a discharge diagnosis for hemorrhage, and/or had a discharge diagnosis for a coagulation or hemorrhagic disorder. Thus, blood loss due to phlebotomy, 1 of the more modifiable etiologies of HAA, was unlikely to have been the primary driver for most patients who developed severe HAA. Since it has been estimated to take 15 days of daily phlebotomy of 53 mL of whole blood in females of average body weight (and 20 days for average weight males) with no bone marrow synthesis for severe anemia to develop, it is even less likely that phlebotomy was the principal etiology given an 8-day median LOS among patients with severe HAA.^19,20 However, since the etiology of HAA can be multifactorial, limiting blood loss due to phlebotomy by using smaller volume tubes, blood conservation devices, or reducing unnecessary testing may mitigate the development of severe HAA.^21,22 Additionally, since more than three-quarters of patients who developed severe HAA underwent a major procedure, more care and attention to minimizing operative blood loss could lessen the severity of HAA and facilitate better recovery. If minimizing blood loss is not feasible, in the absence of symptoms related to anemia or ongoing blood loss, randomized controlled trials overwhelmingly support a restrictive transfusion strategy using a hemoglobin value threshold of 7 mg/dL, even in the postoperative setting.^23-25

The implications of mild to moderate HAA are less clear. The odds ratios for mild and moderate HAA, while not statistically significant, suggest a small increase in harm compared to not developing HAA. Furthermore, the upper boundary of the confidence intervals for mild and moderate HAA cannot exclude a possible 30% and 56% increase in the odds for the 30-day composite outcome, respectively. Thus, a better powered study, including more patients and extending the time interval for ascertaining postdischarge adverse events beyond 30 days, may reveal a harmful association. Lastly, our study assessed only the association of HAA with 30-day readmission and mortality. Examining the association between HAA and other patient-centered outcomes such as fatigue, functional impairment, and prolonged posthospitalization recovery time may uncover other important adverse effects of mild and moderate HAA, both of which occur far more frequently than severe HAA.

Our findings should be interpreted in the context of several limitations. First, although we included a diverse group of patients from a multihospital cohort, generalizability to other settings is uncertain. Second, as this was a retrospective study using EHR data, we had limited information to infer the precise mechanism of HAA for each patient. However, procedure codes and discharge diagnoses enabled us to assess which patients underwent a major procedure or had a hemorrhage or hemorrhagic disorder during the hospitalization. Third, given the relatively few number of patients with severe HAA in our cohort, we were unable to assess if the association of severe HAA differed by suspected etiology. Lastly, because we were unable to ascertain the timing of the hematocrit values within the first 24 hours of admission, we excluded both patients with preexisting anemia on admission and those who developed HAA within the first 24 hours of admission, which is not uncommon.²⁶ Thus, we were unable to assess the effect of acute on chronic anemia arising during hospitalization and HAA that develops within the first 24 hours, both of which may also be harmful.^18,27,28

In conclusion, severe HAA occurs in 1.4% of all medical hospitalizations and is associated with increased odds of death or readmission within 30 days. Since most patients with severe HAA had undergone a major procedure or had a discharge diagnosis of hemorrhage or a coagulation or hemorrhagic disorder, it is unclear if severe HAA is potentially preventable through preventing blood loss from phlebotomy or by reducing iatrogenic injury during procedures. Future research should assess the potential preventability of severe HAA, and examine other patient-centered outcomes potentially related to anemia, including fatigue, functional impairment, and trajectory of posthospital recovery.

Acknowledgments

The authors would like to acknowledge Ruben Amarasingham, MD, MBA, President and Chief Executive Officer of the Parkland Center for Clinical Innovation, and Ferdinand Velasco, MD, Chief Health Information Officer at Texas Health Resources, for their assistance in assembling the 6 hospital cohort used in this study. The authors would also like to thank Valy Fontil, MD, MAS, Assistant Professor of Medicine at the University of California San Francisco School of Medicine, and Elizabeth Rogers, MD, MAS, Assistant Professor of Internal Medicine and Pediatrics at the University of Minnesota Medical School, for their constructive feedback on an earlier version of this manuscript.

Disclosures

This work was supported by the Agency for Healthcare Research and Quality-funded UT Southwestern Center for Patient-Centered Outcomes Research (R24 HS022418-01); the Commonwealth Foundation (#20100323); the UT Southwestern KL2 Scholars Program supported by the National Institutes of Health (KL2 TR001103); the National Center for Advancing Translational Sciences at the National Institute of Health (U54 RFA-TR-12-006); and the National Institute on Aging (K23AG052603). The study sponsors had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no financial conflicts of interest to disclose.

Hospital-acquired anemia (HAA) is defined as having a normal hemoglobin value upon admission but developing anemia during the course of hospitalization. The condition is common, with an incidence ranging from approximately 25% when defined by using the hemoglobin value prior to discharge to 74% when using the nadir hemoglobin value during hospitalization.^1-5 While there are many potential etiologies for HAA, given that iatrogenic blood loss from phlebotomy may lead to its development,^6,7 HAA has been postulated to be a hazard of hospitalization that is potentially preventable.⁸ However, it is unclear whether the development of HAA portends worse outcomes after hospital discharge.

The limited number of studies on the association between HAA and postdischarge outcomes has been restricted to patients hospitalized for acute myocardial infarction (AMI).^3,9,10 Among this subpopulation, HAA is independently associated with greater morbidity and mortality following hospital discharge.^3,9,10 In a more broadly representative population of hospitalized adults, Koch et al.² found that the development of HAA is associated with greater length of stay (LOS), hospital charges, and inpatient mortality. However, given that HAA was defined by the lowest hemoglobin level during hospitalization (and not necessarily the last value prior to discharge), it is unclear if the worse outcomes observed were the cause of the HAA, rather than its effect, since hospital LOS is a robust predictor for the development of HAA, as well as a major driver of hospital costs and a prognostic marker for inpatient mortality.^3,9 Furthermore, this study evaluated outcomes only during the index hospitalization, so it is unclear if patients who develop HAA have worse clinical outcomes after discharge.

Therefore, in this study, we used clinically granular electronic health record (EHR) data from a diverse cohort of consecutive medicine inpatients hospitalized for any reason at 1 of 6 hospitals to: 1) describe the epidemiology of HAA; 2) identify predictors of its development; and 3) examine its association with 30-day postdischarge adverse outcomes. We hypothesized that the development of HAA would be independently associated with 30-day readmission and mortality in a dose-dependent fashion, with increasing severity of HAA associated with worse outcomes.

Study Design, Population, and Data Sources

We conducted a retrospective observational cohort study using EHR data collected from November 1, 2009 to October 30, 2010 from 6 hospitals in the north Texas region. One site was a university-affiliated safety-net hospital; the remaining 5 community hospitals were a mix of teaching and nonteaching sites. All hospitals used the Epic EHR system (Epic Systems Corporation, Verona, Wisconsin). Details of this cohort have been published.^11,12This study included consecutive hospitalizations among adults age 18 years or older who were discharged from a medicine inpatient service with any diagnosis. We excluded hospitalizations by individuals who were anemic within the first 24 hours of admission (hematocrit less than 36% for women and less than 40% for men), were missing a hematocrit value within the first 24 hours of hospitalization or a repeat hematocrit value prior to discharge, had a hospitalization in the preceding 30 days (ie, index hospitalization was considered a readmission), died in the hospital, were transferred to another hospital, or left against medical advice. For individuals with multiple eligible hospitalizations during the study period, we included only the first hospitalization. We also excluded those discharged to hospice, given that this population of individuals may have intentionally desired less aggressive care.

Definition of Hospital-Acquired Anemia

HAA was defined as having a normal hematocrit value (36% or greater for women and 40% or greater for men) within the first 24 hours of admission and a hematocrit value at the time of hospital discharge lower than the World Health Organization’s sex-specific cut points.¹³ If there was more than 1 hematocrit value on the same day, we chose the lowest value. Based on prior studies, HAA was further categorized by severity as mild (hematocrit greater than 33% and less than 36% in women; and greater than 33% and less than 40% in men), moderate (hematocrit greater than 27% and 33% or less for all), or severe (hematocrit 27% or less for all).^2,14

Characteristics

We extracted information on sociodemographic characteristics, comorbidities, LOS, procedures, blood transfusions, and laboratory values from the EHR. Hospitalizations in the 12 months preceding the index hospitalization were ascertained from the EHR and from an all-payer regional hospitalization database that captures hospitalizations from 75 acute care hospitals within a 100-mile radius of Dallas-Fort Worth. International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) discharge diagnosis codes were categorized according to the Agency for Healthcare Research and Quality (AHRQ) Clinical Classifications Software (CCS).¹⁵ We defined a diagnosis for hemorrhage and coagulation, and hemorrhagic disorder as the presence of any ICD-9-CM code (primary or secondary) that mapped to the AHRQ CCS diagnoses 60 and 153, and 62, respectively. Procedures were categorized as minor diagnostic, minor therapeutic, major diagnostic, and major therapeutic using the AHRQ Healthcare Cost and Utilization Procedure Classes tool.¹⁶

Outcomes

The primary outcome was a composite of death or readmission within 30 days of hospital discharge. Hospital readmissions were ascertained at the index hospital and at any of 75 acute care hospitals in the region as described earlier. Death was ascertained from each of the hospitals’ EHR and administrative data and the Social Security Death Index. Individuals who had both outcomes (eg, a 30-day readmission and death) were considered to have only 1 occurrence of the composite primary outcome measure. Our secondary outcomes were death and readmission within 30 days of discharge, considered as separate outcomes.

Statistical Analysis

We used logistic regression models to evaluate predictors of HAA and to estimate the association of HAA on subsequent 30-day adverse outcomes after hospital discharge. All models accounted for clustering of patients by hospital. For the outcomes analyses, models were adjusted for potential confounders based on prior literature and our group’s expertise, which included age, sex, race/ethnicity, Charlson comorbidity index, prior hospitalizations, nonelective admission status, creatinine level on admission, blood urea nitrogen (BUN) to creatinine ratio of more than 20:1 on admission, LOS, receipt of a major diagnostic or therapeutic procedure during the index hospitalization, a discharge diagnosis for hemorrhage, and a discharge diagnosis for a coagulation or hemorrhagic disorder. For the mortality analyses, given the limited number of 30-day deaths after hospital discharge in our cohort, we collapsed moderate and severe HAA into a single category. In sensitivity analyses, we repeated the adjusted model, but excluded patients in our cohort who had received at least 1 blood transfusion during the index hospitalization (2.6%) given its potential for harm, and patients with a primary discharge diagnosis for AMI (3.1%).¹⁷

The functional forms of continuous variables were assessed using restricted cubic splines and locally weighted scatterplot smoothing techniques. All analyses were performed using STATA statistical software version 12.0 (StataCorp, College Station, Texas). The University of Texas Southwestern Medical Center institutional review board approved this study.

Of 53,995 consecutive medicine hospitalizations among adults age 18 years or older during our study period, 11,309 index hospitalizations were included in our study cohort (Supplemental Figure 1). The majority of patients excluded were because of having documented anemia within the first 24 hours of admission (n=24,950). With increasing severity of HAA, patients were older, more likely to be female, non-Hispanic white, electively admitted, have fewer comorbidities, less likely to be hospitalized in the past year, more likely to have had a major procedure, receive a blood transfusion, have a longer LOS, and have a primary or secondary discharge diagnosis for a hemorrhage or a coagulation or hemorrhagic disorder (Table 1).

Epidemiology of HAA

Among this cohort of patients without anemia on admission, the median hematocrit value on admission was 40.6 g/dL and on discharge was 38.9 g/dL. One-third of patients with normal hematocrit value at admission developed HAA, with 21.6% developing mild HAA, 10.1% developing moderate HAA, and 1.4% developing severe HAA. The median discharge hematocrit value was 36 g/dL (interquartile range [IQR]), 35-38 g/dL) for the group of patients who developed mild HAA, 31 g/dL (IQR, 30-32 g/dL) for moderate HAA, and 26 g/dL (IQR, 25-27 g/dL) for severe HAA (Supplemental Figure 2). Among the severe HAA group, 135 of the 159 patients (85%) had a major procedure (n=123, accounting for 219 unique major procedures), a diagnosis for hemorrhage (n=30), and/or a diagnosis for a coagulation or hemorrhagic disorder (n=23) during the index hospitalization. Of the 219 major procedures among patients with severe HAA, most were musculoskeletal (92 procedures), cardiovascular (61 procedures), or digestive system-related (41 procedures). The most common types of procedures were coronary artery bypass graft (36 procedures), hip replacement (25 procedures), knee replacement (17 procedures), and femur fracture reduction (15 procedures). The 10 most common principal discharge diagnoses of the index hospitalization by HAA group are shown in Supplemental Table 1. For the severe HAA group, the most common diagnosis was hip fracture (20.8%).

Predictors of HAA

Compared to no or mild HAA, female sex, elective admission status, serum creatinine on admission, BUN to creatinine ratio greater than 20 to 1, hospital LOS, and undergoing a major diagnostic or therapeutic procedure were predictors for the development of moderate or severe HAA (Table 2). The model explained 23% of the variance (McFadden’s pseudo R²).

Incidence of Postdischarge Outcomes by Severity of HAA

The severity of HAA was associated with a dose-dependent increase in the incidence of 30-day adverse outcomes, such that patients with increasing severity of HAA had greater 30-day composite, mortality, and readmission outcomes (P < 0.001; Figure). The 30-day postdischarge composite outcome was primarily driven by hospital readmissions given the low mortality rate in our cohort. Patients who did not develop HAA had an incidence of 9.7% for the composite outcome, whereas patients with severe HAA had an incidence of 16.4%. Among the 24 patients with severe HAA but who had not undergone a major procedure or had a discharge diagnosis for hemorrhage or for a coagulation or hemorrhagic disorder, only 3 (12.5%) had a composite postdischarge adverse outcome (2 readmissions and 1 death). The median time to readmission was similar between groups, but more patients with severe HAA had an early readmission within 7 days of hospital discharge than patients who did not develop HAA (6.9% vs. 2.9%, P = 0.001; Supplemental Table 2).

Association of HAA and Postdischarge Outcomes

In unadjusted analyses, compared to not developing HAA, mild, moderate, and severe HAA were associated with a 29%, 61%, and 81% increase in the odds for a composite outcome, respectively (Table 3). After adjustment for confounders, the effect size for HAA attenuated and was no longer statistically significant for mild and moderate HAA. However, severe HAA was significantly associated with a 39% increase in the odds for the composite outcome and a 41% increase in the odds for 30-day readmission (P = 0.008 and P = 0.02, respectively).

In sensitivity analyses, the exclusion of individuals who received at least 1 blood transfusion during the index hospitalization (n=298) and individuals who had a primary discharge diagnosis for AMI (n=353) did not substantively change the estimates of the association between severe HAA and postdischarge outcomes (Supplemental Tables 3 and 4). However, because of the fewer number of adverse events for each analysis, the confidence intervals were wider and the association of severe HAA and the composite outcome and readmission were no longer statistically significant in these subcohorts.

In this large and diverse sample of medical inpatients, we found that HAA occurs in one-third of adults with normal hematocrit value at admission, where 10.1% of the cohort developed moderately severe HAA and 1.4% developed severe HAA by the time of discharge. Length of stay and undergoing a major diagnostic or therapeutic procedure were the 2 strongest potentially modifiable predictors of developing moderate or severe HAA. Severe HAA was independently associated with a 39% increase in the odds of being readmitted or dying within 30 days after hospital discharge compared to not developing HAA. However, the associations between mild or moderate HAA with adverse outcomes were attenuated after adjusting for confounders and were no longer statistically significant.

To our knowledge, this is the first study on the postdischarge adverse outcomes of HAA among a diverse cohort of medical inpatients hospitalized for any reason. In a more restricted population, Salisbury et al.³ found that patients hospitalized for AMI who developed moderate to severe HAA (hemoglobin value at discharge of 11 g/dL or less) had greater 1-year mortality than those without HAA (8.4% vs. 2.6%, P < 0.001), and had an 82% increase in the hazard for mortality (95% confidence interval, hazard ratio 1.11-2.98). Others have similarly shown that HAA is common among patients hospitalized with AMI and is associated with greater mortality.^5,9,18 Our study extends upon this prior research by showing that severe HAA increases the risk for adverse outcomes for all adult inpatients, not only those hospitalized for AMI or among those receiving blood transfusions.

Despite the increased harm associated with severe HAA, it is unclear whether HAA is a preventable hazard of hospitalization, as suggested by others.^6,8 Most patients in our cohort who developed severe HAA underwent a major procedure, had a discharge diagnosis for hemorrhage, and/or had a discharge diagnosis for a coagulation or hemorrhagic disorder. Thus, blood loss due to phlebotomy, 1 of the more modifiable etiologies of HAA, was unlikely to have been the primary driver for most patients who developed severe HAA. Since it has been estimated to take 15 days of daily phlebotomy of 53 mL of whole blood in females of average body weight (and 20 days for average weight males) with no bone marrow synthesis for severe anemia to develop, it is even less likely that phlebotomy was the principal etiology given an 8-day median LOS among patients with severe HAA.^19,20 However, since the etiology of HAA can be multifactorial, limiting blood loss due to phlebotomy by using smaller volume tubes, blood conservation devices, or reducing unnecessary testing may mitigate the development of severe HAA.^21,22 Additionally, since more than three-quarters of patients who developed severe HAA underwent a major procedure, more care and attention to minimizing operative blood loss could lessen the severity of HAA and facilitate better recovery. If minimizing blood loss is not feasible, in the absence of symptoms related to anemia or ongoing blood loss, randomized controlled trials overwhelmingly support a restrictive transfusion strategy using a hemoglobin value threshold of 7 mg/dL, even in the postoperative setting.^23-25

The implications of mild to moderate HAA are less clear. The odds ratios for mild and moderate HAA, while not statistically significant, suggest a small increase in harm compared to not developing HAA. Furthermore, the upper boundary of the confidence intervals for mild and moderate HAA cannot exclude a possible 30% and 56% increase in the odds for the 30-day composite outcome, respectively. Thus, a better powered study, including more patients and extending the time interval for ascertaining postdischarge adverse events beyond 30 days, may reveal a harmful association. Lastly, our study assessed only the association of HAA with 30-day readmission and mortality. Examining the association between HAA and other patient-centered outcomes such as fatigue, functional impairment, and prolonged posthospitalization recovery time may uncover other important adverse effects of mild and moderate HAA, both of which occur far more frequently than severe HAA.

Our findings should be interpreted in the context of several limitations. First, although we included a diverse group of patients from a multihospital cohort, generalizability to other settings is uncertain. Second, as this was a retrospective study using EHR data, we had limited information to infer the precise mechanism of HAA for each patient. However, procedure codes and discharge diagnoses enabled us to assess which patients underwent a major procedure or had a hemorrhage or hemorrhagic disorder during the hospitalization. Third, given the relatively few number of patients with severe HAA in our cohort, we were unable to assess if the association of severe HAA differed by suspected etiology. Lastly, because we were unable to ascertain the timing of the hematocrit values within the first 24 hours of admission, we excluded both patients with preexisting anemia on admission and those who developed HAA within the first 24 hours of admission, which is not uncommon.²⁶ Thus, we were unable to assess the effect of acute on chronic anemia arising during hospitalization and HAA that develops within the first 24 hours, both of which may also be harmful.^18,27,28

In conclusion, severe HAA occurs in 1.4% of all medical hospitalizations and is associated with increased odds of death or readmission within 30 days. Since most patients with severe HAA had undergone a major procedure or had a discharge diagnosis of hemorrhage or a coagulation or hemorrhagic disorder, it is unclear if severe HAA is potentially preventable through preventing blood loss from phlebotomy or by reducing iatrogenic injury during procedures. Future research should assess the potential preventability of severe HAA, and examine other patient-centered outcomes potentially related to anemia, including fatigue, functional impairment, and trajectory of posthospital recovery.

Acknowledgments

The authors would like to acknowledge Ruben Amarasingham, MD, MBA, President and Chief Executive Officer of the Parkland Center for Clinical Innovation, and Ferdinand Velasco, MD, Chief Health Information Officer at Texas Health Resources, for their assistance in assembling the 6 hospital cohort used in this study. The authors would also like to thank Valy Fontil, MD, MAS, Assistant Professor of Medicine at the University of California San Francisco School of Medicine, and Elizabeth Rogers, MD, MAS, Assistant Professor of Internal Medicine and Pediatrics at the University of Minnesota Medical School, for their constructive feedback on an earlier version of this manuscript.

Disclosures

This work was supported by the Agency for Healthcare Research and Quality-funded UT Southwestern Center for Patient-Centered Outcomes Research (R24 HS022418-01); the Commonwealth Foundation (#20100323); the UT Southwestern KL2 Scholars Program supported by the National Institutes of Health (KL2 TR001103); the National Center for Advancing Translational Sciences at the National Institute of Health (U54 RFA-TR-12-006); and the National Institute on Aging (K23AG052603). The study sponsors had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no financial conflicts of interest to disclose.

Hospital-acquired anemia (HAA) is defined as having a normal hemoglobin value upon admission but developing anemia during the course of hospitalization. The condition is common, with an incidence ranging from approximately 25% when defined by using the hemoglobin value prior to discharge to 74% when using the nadir hemoglobin value during hospitalization.^1-5 While there are many potential etiologies for HAA, given that iatrogenic blood loss from phlebotomy may lead to its development,^6,7 HAA has been postulated to be a hazard of hospitalization that is potentially preventable.⁸ However, it is unclear whether the development of HAA portends worse outcomes after hospital discharge.

The limited number of studies on the association between HAA and postdischarge outcomes has been restricted to patients hospitalized for acute myocardial infarction (AMI).^3,9,10 Among this subpopulation, HAA is independently associated with greater morbidity and mortality following hospital discharge.^3,9,10 In a more broadly representative population of hospitalized adults, Koch et al.² found that the development of HAA is associated with greater length of stay (LOS), hospital charges, and inpatient mortality. However, given that HAA was defined by the lowest hemoglobin level during hospitalization (and not necessarily the last value prior to discharge), it is unclear if the worse outcomes observed were the cause of the HAA, rather than its effect, since hospital LOS is a robust predictor for the development of HAA, as well as a major driver of hospital costs and a prognostic marker for inpatient mortality.^3,9 Furthermore, this study evaluated outcomes only during the index hospitalization, so it is unclear if patients who develop HAA have worse clinical outcomes after discharge.

Therefore, in this study, we used clinically granular electronic health record (EHR) data from a diverse cohort of consecutive medicine inpatients hospitalized for any reason at 1 of 6 hospitals to: 1) describe the epidemiology of HAA; 2) identify predictors of its development; and 3) examine its association with 30-day postdischarge adverse outcomes. We hypothesized that the development of HAA would be independently associated with 30-day readmission and mortality in a dose-dependent fashion, with increasing severity of HAA associated with worse outcomes.

Study Design, Population, and Data Sources

We conducted a retrospective observational cohort study using EHR data collected from November 1, 2009 to October 30, 2010 from 6 hospitals in the north Texas region. One site was a university-affiliated safety-net hospital; the remaining 5 community hospitals were a mix of teaching and nonteaching sites. All hospitals used the Epic EHR system (Epic Systems Corporation, Verona, Wisconsin). Details of this cohort have been published.^11,12This study included consecutive hospitalizations among adults age 18 years or older who were discharged from a medicine inpatient service with any diagnosis. We excluded hospitalizations by individuals who were anemic within the first 24 hours of admission (hematocrit less than 36% for women and less than 40% for men), were missing a hematocrit value within the first 24 hours of hospitalization or a repeat hematocrit value prior to discharge, had a hospitalization in the preceding 30 days (ie, index hospitalization was considered a readmission), died in the hospital, were transferred to another hospital, or left against medical advice. For individuals with multiple eligible hospitalizations during the study period, we included only the first hospitalization. We also excluded those discharged to hospice, given that this population of individuals may have intentionally desired less aggressive care.

Definition of Hospital-Acquired Anemia

HAA was defined as having a normal hematocrit value (36% or greater for women and 40% or greater for men) within the first 24 hours of admission and a hematocrit value at the time of hospital discharge lower than the World Health Organization’s sex-specific cut points.¹³ If there was more than 1 hematocrit value on the same day, we chose the lowest value. Based on prior studies, HAA was further categorized by severity as mild (hematocrit greater than 33% and less than 36% in women; and greater than 33% and less than 40% in men), moderate (hematocrit greater than 27% and 33% or less for all), or severe (hematocrit 27% or less for all).^2,14

Characteristics

We extracted information on sociodemographic characteristics, comorbidities, LOS, procedures, blood transfusions, and laboratory values from the EHR. Hospitalizations in the 12 months preceding the index hospitalization were ascertained from the EHR and from an all-payer regional hospitalization database that captures hospitalizations from 75 acute care hospitals within a 100-mile radius of Dallas-Fort Worth. International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) discharge diagnosis codes were categorized according to the Agency for Healthcare Research and Quality (AHRQ) Clinical Classifications Software (CCS).¹⁵ We defined a diagnosis for hemorrhage and coagulation, and hemorrhagic disorder as the presence of any ICD-9-CM code (primary or secondary) that mapped to the AHRQ CCS diagnoses 60 and 153, and 62, respectively. Procedures were categorized as minor diagnostic, minor therapeutic, major diagnostic, and major therapeutic using the AHRQ Healthcare Cost and Utilization Procedure Classes tool.¹⁶

Outcomes

The primary outcome was a composite of death or readmission within 30 days of hospital discharge. Hospital readmissions were ascertained at the index hospital and at any of 75 acute care hospitals in the region as described earlier. Death was ascertained from each of the hospitals’ EHR and administrative data and the Social Security Death Index. Individuals who had both outcomes (eg, a 30-day readmission and death) were considered to have only 1 occurrence of the composite primary outcome measure. Our secondary outcomes were death and readmission within 30 days of discharge, considered as separate outcomes.

Statistical Analysis

We used logistic regression models to evaluate predictors of HAA and to estimate the association of HAA on subsequent 30-day adverse outcomes after hospital discharge. All models accounted for clustering of patients by hospital. For the outcomes analyses, models were adjusted for potential confounders based on prior literature and our group’s expertise, which included age, sex, race/ethnicity, Charlson comorbidity index, prior hospitalizations, nonelective admission status, creatinine level on admission, blood urea nitrogen (BUN) to creatinine ratio of more than 20:1 on admission, LOS, receipt of a major diagnostic or therapeutic procedure during the index hospitalization, a discharge diagnosis for hemorrhage, and a discharge diagnosis for a coagulation or hemorrhagic disorder. For the mortality analyses, given the limited number of 30-day deaths after hospital discharge in our cohort, we collapsed moderate and severe HAA into a single category. In sensitivity analyses, we repeated the adjusted model, but excluded patients in our cohort who had received at least 1 blood transfusion during the index hospitalization (2.6%) given its potential for harm, and patients with a primary discharge diagnosis for AMI (3.1%).¹⁷

The functional forms of continuous variables were assessed using restricted cubic splines and locally weighted scatterplot smoothing techniques. All analyses were performed using STATA statistical software version 12.0 (StataCorp, College Station, Texas). The University of Texas Southwestern Medical Center institutional review board approved this study.

Of 53,995 consecutive medicine hospitalizations among adults age 18 years or older during our study period, 11,309 index hospitalizations were included in our study cohort (Supplemental Figure 1). The majority of patients excluded were because of having documented anemia within the first 24 hours of admission (n=24,950). With increasing severity of HAA, patients were older, more likely to be female, non-Hispanic white, electively admitted, have fewer comorbidities, less likely to be hospitalized in the past year, more likely to have had a major procedure, receive a blood transfusion, have a longer LOS, and have a primary or secondary discharge diagnosis for a hemorrhage or a coagulation or hemorrhagic disorder (Table 1).

Epidemiology of HAA

Among this cohort of patients without anemia on admission, the median hematocrit value on admission was 40.6 g/dL and on discharge was 38.9 g/dL. One-third of patients with normal hematocrit value at admission developed HAA, with 21.6% developing mild HAA, 10.1% developing moderate HAA, and 1.4% developing severe HAA. The median discharge hematocrit value was 36 g/dL (interquartile range [IQR]), 35-38 g/dL) for the group of patients who developed mild HAA, 31 g/dL (IQR, 30-32 g/dL) for moderate HAA, and 26 g/dL (IQR, 25-27 g/dL) for severe HAA (Supplemental Figure 2). Among the severe HAA group, 135 of the 159 patients (85%) had a major procedure (n=123, accounting for 219 unique major procedures), a diagnosis for hemorrhage (n=30), and/or a diagnosis for a coagulation or hemorrhagic disorder (n=23) during the index hospitalization. Of the 219 major procedures among patients with severe HAA, most were musculoskeletal (92 procedures), cardiovascular (61 procedures), or digestive system-related (41 procedures). The most common types of procedures were coronary artery bypass graft (36 procedures), hip replacement (25 procedures), knee replacement (17 procedures), and femur fracture reduction (15 procedures). The 10 most common principal discharge diagnoses of the index hospitalization by HAA group are shown in Supplemental Table 1. For the severe HAA group, the most common diagnosis was hip fracture (20.8%).

Predictors of HAA

Compared to no or mild HAA, female sex, elective admission status, serum creatinine on admission, BUN to creatinine ratio greater than 20 to 1, hospital LOS, and undergoing a major diagnostic or therapeutic procedure were predictors for the development of moderate or severe HAA (Table 2). The model explained 23% of the variance (McFadden’s pseudo R²).

Incidence of Postdischarge Outcomes by Severity of HAA

The severity of HAA was associated with a dose-dependent increase in the incidence of 30-day adverse outcomes, such that patients with increasing severity of HAA had greater 30-day composite, mortality, and readmission outcomes (P < 0.001; Figure). The 30-day postdischarge composite outcome was primarily driven by hospital readmissions given the low mortality rate in our cohort. Patients who did not develop HAA had an incidence of 9.7% for the composite outcome, whereas patients with severe HAA had an incidence of 16.4%. Among the 24 patients with severe HAA but who had not undergone a major procedure or had a discharge diagnosis for hemorrhage or for a coagulation or hemorrhagic disorder, only 3 (12.5%) had a composite postdischarge adverse outcome (2 readmissions and 1 death). The median time to readmission was similar between groups, but more patients with severe HAA had an early readmission within 7 days of hospital discharge than patients who did not develop HAA (6.9% vs. 2.9%, P = 0.001; Supplemental Table 2).

Association of HAA and Postdischarge Outcomes

In unadjusted analyses, compared to not developing HAA, mild, moderate, and severe HAA were associated with a 29%, 61%, and 81% increase in the odds for a composite outcome, respectively (Table 3). After adjustment for confounders, the effect size for HAA attenuated and was no longer statistically significant for mild and moderate HAA. However, severe HAA was significantly associated with a 39% increase in the odds for the composite outcome and a 41% increase in the odds for 30-day readmission (P = 0.008 and P = 0.02, respectively).

In sensitivity analyses, the exclusion of individuals who received at least 1 blood transfusion during the index hospitalization (n=298) and individuals who had a primary discharge diagnosis for AMI (n=353) did not substantively change the estimates of the association between severe HAA and postdischarge outcomes (Supplemental Tables 3 and 4). However, because of the fewer number of adverse events for each analysis, the confidence intervals were wider and the association of severe HAA and the composite outcome and readmission were no longer statistically significant in these subcohorts.

In this large and diverse sample of medical inpatients, we found that HAA occurs in one-third of adults with normal hematocrit value at admission, where 10.1% of the cohort developed moderately severe HAA and 1.4% developed severe HAA by the time of discharge. Length of stay and undergoing a major diagnostic or therapeutic procedure were the 2 strongest potentially modifiable predictors of developing moderate or severe HAA. Severe HAA was independently associated with a 39% increase in the odds of being readmitted or dying within 30 days after hospital discharge compared to not developing HAA. However, the associations between mild or moderate HAA with adverse outcomes were attenuated after adjusting for confounders and were no longer statistically significant.

To our knowledge, this is the first study on the postdischarge adverse outcomes of HAA among a diverse cohort of medical inpatients hospitalized for any reason. In a more restricted population, Salisbury et al.³ found that patients hospitalized for AMI who developed moderate to severe HAA (hemoglobin value at discharge of 11 g/dL or less) had greater 1-year mortality than those without HAA (8.4% vs. 2.6%, P < 0.001), and had an 82% increase in the hazard for mortality (95% confidence interval, hazard ratio 1.11-2.98). Others have similarly shown that HAA is common among patients hospitalized with AMI and is associated with greater mortality.^5,9,18 Our study extends upon this prior research by showing that severe HAA increases the risk for adverse outcomes for all adult inpatients, not only those hospitalized for AMI or among those receiving blood transfusions.

Despite the increased harm associated with severe HAA, it is unclear whether HAA is a preventable hazard of hospitalization, as suggested by others.^6,8 Most patients in our cohort who developed severe HAA underwent a major procedure, had a discharge diagnosis for hemorrhage, and/or had a discharge diagnosis for a coagulation or hemorrhagic disorder. Thus, blood loss due to phlebotomy, 1 of the more modifiable etiologies of HAA, was unlikely to have been the primary driver for most patients who developed severe HAA. Since it has been estimated to take 15 days of daily phlebotomy of 53 mL of whole blood in females of average body weight (and 20 days for average weight males) with no bone marrow synthesis for severe anemia to develop, it is even less likely that phlebotomy was the principal etiology given an 8-day median LOS among patients with severe HAA.^19,20 However, since the etiology of HAA can be multifactorial, limiting blood loss due to phlebotomy by using smaller volume tubes, blood conservation devices, or reducing unnecessary testing may mitigate the development of severe HAA.^21,22 Additionally, since more than three-quarters of patients who developed severe HAA underwent a major procedure, more care and attention to minimizing operative blood loss could lessen the severity of HAA and facilitate better recovery. If minimizing blood loss is not feasible, in the absence of symptoms related to anemia or ongoing blood loss, randomized controlled trials overwhelmingly support a restrictive transfusion strategy using a hemoglobin value threshold of 7 mg/dL, even in the postoperative setting.^23-25

The implications of mild to moderate HAA are less clear. The odds ratios for mild and moderate HAA, while not statistically significant, suggest a small increase in harm compared to not developing HAA. Furthermore, the upper boundary of the confidence intervals for mild and moderate HAA cannot exclude a possible 30% and 56% increase in the odds for the 30-day composite outcome, respectively. Thus, a better powered study, including more patients and extending the time interval for ascertaining postdischarge adverse events beyond 30 days, may reveal a harmful association. Lastly, our study assessed only the association of HAA with 30-day readmission and mortality. Examining the association between HAA and other patient-centered outcomes such as fatigue, functional impairment, and prolonged posthospitalization recovery time may uncover other important adverse effects of mild and moderate HAA, both of which occur far more frequently than severe HAA.

Our findings should be interpreted in the context of several limitations. First, although we included a diverse group of patients from a multihospital cohort, generalizability to other settings is uncertain. Second, as this was a retrospective study using EHR data, we had limited information to infer the precise mechanism of HAA for each patient. However, procedure codes and discharge diagnoses enabled us to assess which patients underwent a major procedure or had a hemorrhage or hemorrhagic disorder during the hospitalization. Third, given the relatively few number of patients with severe HAA in our cohort, we were unable to assess if the association of severe HAA differed by suspected etiology. Lastly, because we were unable to ascertain the timing of the hematocrit values within the first 24 hours of admission, we excluded both patients with preexisting anemia on admission and those who developed HAA within the first 24 hours of admission, which is not uncommon.²⁶ Thus, we were unable to assess the effect of acute on chronic anemia arising during hospitalization and HAA that develops within the first 24 hours, both of which may also be harmful.^18,27,28

In conclusion, severe HAA occurs in 1.4% of all medical hospitalizations and is associated with increased odds of death or readmission within 30 days. Since most patients with severe HAA had undergone a major procedure or had a discharge diagnosis of hemorrhage or a coagulation or hemorrhagic disorder, it is unclear if severe HAA is potentially preventable through preventing blood loss from phlebotomy or by reducing iatrogenic injury during procedures. Future research should assess the potential preventability of severe HAA, and examine other patient-centered outcomes potentially related to anemia, including fatigue, functional impairment, and trajectory of posthospital recovery.

Acknowledgments

The authors would like to acknowledge Ruben Amarasingham, MD, MBA, President and Chief Executive Officer of the Parkland Center for Clinical Innovation, and Ferdinand Velasco, MD, Chief Health Information Officer at Texas Health Resources, for their assistance in assembling the 6 hospital cohort used in this study. The authors would also like to thank Valy Fontil, MD, MAS, Assistant Professor of Medicine at the University of California San Francisco School of Medicine, and Elizabeth Rogers, MD, MAS, Assistant Professor of Internal Medicine and Pediatrics at the University of Minnesota Medical School, for their constructive feedback on an earlier version of this manuscript.

Disclosures

This work was supported by the Agency for Healthcare Research and Quality-funded UT Southwestern Center for Patient-Centered Outcomes Research (R24 HS022418-01); the Commonwealth Foundation (#20100323); the UT Southwestern KL2 Scholars Program supported by the National Institutes of Health (KL2 TR001103); the National Center for Advancing Translational Sciences at the National Institute of Health (U54 RFA-TR-12-006); and the National Institute on Aging (K23AG052603). The study sponsors had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no financial conflicts of interest to disclose.

Pneumonia is a leading cause of hospitalizations in the U.S., accounting for more than 1.1 million discharges annually.¹ Pneumonia is frequently complicated by hospital readmission, which is costly and potentially avoidable.^2,3 Due to financial penalties imposed on hospitals for higher than expected 30-day readmission rates, there is increasing attention to implementing interventions to reduce readmissions in this population.^4,5 However, because these programs are resource-intensive, interventions are thought to be most cost-effective if they are targeted to high-risk individuals who are most likely to benefit.^6-8

Current pneumonia-specific readmission risk-prediction models that could enable identification of high-risk patients suffer from poor predictive ability, greatly limiting their use, and most were validated among older adults or by using data from single academic medical centers, limiting their generalizability.^9-14 A potential reason for poor predictive accuracy is the omission of known robust clinical predictors of pneumonia-related outcomes, including pneumonia severity of illness and stability on discharge.^15-17 Approaches using electronic health record (EHR) data, which include this clinically granular data, could enable hospitals to more accurately and pragmatically identify high-risk patients during the index hospitalization and enable interventions to be initiated prior to discharge.

An alternative strategy to identifying high-risk patients for readmission is to use a multi-condition risk-prediction model. Developing and implementing models for every condition may be time-consuming and costly. We have derived and validated 2 multi-condition risk-prediction models using EHR data—1 using data from the first day of hospital admission (‘first-day’ model), and the second incorporating data from the entire hospitalization (‘full-stay’ model) to reflect in-hospital complications and clinical stability at discharge.^18,19 However, it is unknown if a multi-condition model for pneumonia would perform as well as a disease-specific model.

This study aimed to develop 2 EHR-based pneumonia-specific readmission risk-prediction models using data routinely collected in clinical practice—a ‘first-day’ and a ‘full-stay’ model—and compare the performance of each model to: 1) one another; 2) the corresponding multi-condition EHR model; and 3) to other potentially useful models in predicting pneumonia readmissions (the Centers for Medicare and Medicaid Services [CMS] pneumonia model, and 2 commonly used pneumonia severity of illness scores validated for predicting mortality). We hypothesized that the pneumonia-specific EHR models would outperform other models; and the full-stay pneumonia-specific model would outperform the first-day pneumonia-specific model.

Study Design, Population, and Data Sources

We conducted an observational study using EHR data collected from 6 hospitals (including safety net, community, teaching, and nonteaching hospitals) in north Texas between November 2009 and October 2010, All hospitals used the Epic EHR (Epic Systems Corporation, Verona, WI). Details of this cohort have been published.^18,19

We included consecutive hospitalizations among adults 18 years and older discharged from any medicine service with principal discharge diagnoses of pneumonia (ICD-9-CM codes 480-483, 485, 486-487), sepsis (ICD-9-CM codes 038, 995.91, 995.92, 785.52), or respiratory failure (ICD-9-CM codes 518.81, 518.82, 518.84, 799.1) when the latter 2 were also accompanied by a secondary diagnosis of pneumonia.²⁰ For individuals with multiple hospitalizations during the study period, we included only the first hospitalization. We excluded individuals who died during the index hospitalization or within 30 days of discharge, were transferred to another acute care facility, or left against medical advice.

Outcomes

The primary outcome was all-cause 30-day readmission, defined as a nonelective hospitalization within 30 days of discharge to any of 75 acute care hospitals within a 100-mile radius of Dallas, ascertained from an all-payer regional hospitalization database.

Predictor Variables for the Pneumonia-Specific Readmission Models

The selection of candidate predictors was informed by our validated multi-condition risk-prediction models using EHR data available within 24 hours of admission (‘first-day’ multi-condition EHR model) or during the entire hospitalization (‘full-stay’ multi-condition EHR model).^18,19 For the pneumonia-specific models, we included all variables in our published multi-condition models as candidate predictors, including sociodemographics, prior utilization, Charlson Comorbidity Index, select laboratory and vital sign abnormalities, length of stay, hospital complications (eg, venous thromboembolism), vital sign instabilities, and disposition status (see Supplemental Table 1 for complete list of variables). We also assessed additional variables specific to pneumonia for inclusion that were: (1) available in the EHR of all participating hospitals; (2) routinely collected or available at the time of admission or discharge; and (3) plausible predictors of adverse outcomes based on literature and clinical expertise. These included select comorbidities (eg, psychiatric conditions, chronic lung disease, history of pneumonia),^10,11,21,22 the pneumonia severity index (PSI),^16,23,24 intensive care unit stay, and receipt of invasive or noninvasive ventilation. We used a modified PSI score because certain data elements were missing. The modified PSI (henceforth referred to as PSI) did not include nursing home residence and included diagnostic codes as proxies for the presence of pleural effusion (ICD-9-CM codes 510, 511.1, and 511.9) and altered mental status (ICD-9-CM codes 780.0X, 780.97, 293.0, 293.1, and 348.3X).

Statistical Analysis

Model Derivation. Candidate predictor variables were classified as available in the EHR within 24 hours of admission and/or at the time of discharge. For example, socioeconomic factors could be ascertained within the first day of hospitalization, whereas length of stay would not be available until the day of discharge. Predictors with missing values were assumed to be normal (less than 1% missing for each variable). Univariate relationships between readmission and each candidate predictor were assessed in the overall cohort using a pre-specified significance threshold of P ≤ 0.10. Significant variables were entered in the respective first-day and full-stay pneumonia-specific multivariable logistic regression models using stepwise-backward selection with a pre-specified significance threshold of P ≤ 0.05. In sensitivity analyses, we alternately derived our models using stepwise-forward selection, as well as stepwise-backward selection minimizing the Bayesian information criterion and Akaike information criterion separately. These alternate modeling strategies yielded identical predictors to our final models.

Model Validation. Model validation was performed using 5-fold cross-validation, with the overall cohort randomly divided into 5 equal-size subsets.²⁵ For each cycle, 4 subsets were used for training to estimate model coefficients, and the fifth subset was used for validation. This cycle was repeated 5 times with each randomly-divided subset used once as the validation set. We repeated this entire process 50 times and averaged the C statistic estimates to derive an optimism-corrected C statistic. Model calibration was assessed qualitatively by comparing predicted to observed probabilities of readmission by quintiles of predicted risk, and with the Hosmer-Lemeshow goodness-of-fit test.

Comparison to Other Models. The main comparisons of the first-day and full-stay pneumonia-specific EHR model performance were to each other and the corresponding multi-condition EHR model.^18,19 The multi-condition EHR models were separately derived and validated within the larger parent cohort from which this study cohort was derived, and outperformed the CMS all-cause model, the HOSPITAL model, and the LACE index.¹⁹ To further triangulate our findings, given the lack of other rigorously validated pneumonia-specific risk-prediction models for readmission,¹⁴ we compared the pneumonia-specific EHR models to the CMS pneumonia model derived from administrative claims data,¹⁰ and 2 commonly used risk-prediction scores for short-term mortality among patients with community-acquired pneumonia, the PSI and CURB-65 scores.¹⁶ Although derived and validated using patient-level data, the CMS model was developed to benchmark hospitals according to hospital-level readmission rates.¹⁰ The CURB-65 score in this study was also modified to include the same altered mental status diagnostic codes according to the modified PSI as a proxy for “confusion.” Both the PSI and CURB-65 scores were calculated using the most abnormal values within the first 24 hours of admission. The ‘updated’ PSI and the ‘updated’ CURB-65 were calculated using the most abnormal values within 24 hours prior to discharge, or the last known observation prior to discharge if no results were recorded within this time period. A complete list of variables for each of the comparison models are shown in Supplemental Table 1.

We assessed model performance by calculating the C statistic, integrated discrimination index, and net reclassification index (NRI) compared to our pneumonia-specific models. The integrated discrimination index is the difference in the mean predicted probability of readmission between patients who were and were not actually readmitted between 2 models, where more positive values suggest improvement in model performance compared to a reference model.²⁶ The NRI is defined as the sum of the net proportions of correctly reclassified persons with and without the event of interest.²⁷ Here, we calculated a category-based NRI to evaluate the performance of pneumonia-specific models in correctly classifying individuals with and without readmissions into the 2 highest readmission risk quintiles vs the lowest 3 risk quintiles compared to other models.²⁷ This pre-specified cutoff is relevant for hospitals interested in identifying the highest risk individuals for targeted intervention.⁷ Finally, we assessed calibration of comparator models in our cohort by comparing predicted probability to observed probability of readmission by quintiles of risk for each model. We conducted all analyses using Stata 12.1 (StataCorp, College Station, Texas). This study was approved by the University of Texas Southwestern Medical Center Institutional Review Board.

Of 1463 index hospitalizations (Supplemental Figure 1), the 30-day all-cause readmission rate was 13.6%. Individuals with a 30-day readmission had markedly different sociodemographic and clinical characteristics compared to those not readmitted (Table 1; see Supplemental Table 2 for additional clinical characteristics).

Derivation, Validation, and Performance of the Pneumonia-Specific Readmission Risk-Prediction Models

The final first-day pneumonia-specific EHR model included 7 variables, including sociodemographic characteristics; prior hospitalizations; thrombocytosis, and PSI (Table 2). The first-day pneumonia-specific model had adequate discrimination (C statistic, 0.695; optimism-corrected C statistic 0.675, 95% confidence interval [CI], 0.667-0.685; Table 3). It also effectively stratified individuals across a broad range of risk (average predicted decile of risk ranged from 4% to 33%; Table 3) and was well calibrated (Supplemental Table 3).

The final full-stay pneumonia-specific EHR readmission model included 8 predictors, including 3 variables from the first-day model (median income, thrombocytosis, and prior hospitalizations; Table 2). The full-stay pneumonia-specific EHR model also included vital sign instabilities on discharge, updated PSI, and disposition status (ie, being discharged with home health or to a post-acute care facility was associated with greater odds of readmission, and hospice with lower odds). The full-stay pneumonia-specific EHR model had good discrimination (C statistic, 0.731; optimism-corrected C statistic, 0.714; 95% CI, 0.706-0.720), and stratified individuals across a broad range of risk (average predicted decile of risk ranged from 3% to 37%; Table 3), and was also well calibrated (Supplemental Table 3).

First-Day Pneumonia-Specific EHR Model vs First-Day Multi-Condition EHR Model

The first-day pneumonia-specific EHR model outperformed the first-day multi-condition EHR model with better discrimination (P = 0.029) and more correctly classified individuals in the top 2 highest risk quintiles vs the bottom 3 risk quintiles (Table 3, Supplemental Table 4, and Supplemental Figure 2A). With respect to calibration, the first-day multi-condition EHR model overestimated risk among the highest quintile risk group compared to the first-day pneumonia-specific EHR model (Figure 1A, 1B).

Full-Stay Pneumonia-Specific EHR Model vs Other Models

The full-stay pneumonia-specific EHR model comparatively outperformed the corresponding full-stay multi-condition EHR model, as well as the first-day pneumonia-specific EHR model, the CMS pneumonia model, the updated PSI, and the updated CURB-65 (Table 3, Supplemental Table 5, Supplemental Table 6, and Supplemental Figures 2B and 2C). Compared to the full-stay multi-condition and first-day pneumonia-specific EHR models, the full-stay pneumonia-specific EHR model had better discrimination, better reclassification (NRI, 0.09 and 0.08, respectively), and was able to stratify individuals across a broader range of readmission risk (Table 3). It also had better calibration in the highest quintile risk group compared to the full-stay multi-condition EHR model (Figure 1C and 1D).

Updated vs First-Day Modified PSI and CURB-65 Scores

The updated PSI was more strongly predictive of readmission than the PSI calculated on the day of admission (Wald test, 9.83; P = 0.002). Each 10-point increase in the updated PSI was associated with a 22% increased odds of readmission vs an 11% increase for the PSI calculated upon admission (Table 2). The improved predictive ability of the updated PSI and CURB-65 scores was also reflected in the superior discrimination and calibration vs the respective first-day pneumonia severity of illness scores (Table 3).

Using routinely available EHR data from 6 diverse hospitals, we developed 2 pneumonia-specific readmission risk-prediction models that aimed to allow hospitals to identify patients hospitalized with pneumonia at high risk for readmission. Overall, we found that a pneumonia-specific model using EHR data from the entire hospitalization outperformed all other models—including the first-day pneumonia-specific model using data present only on admission, our own multi-condition EHR models, and the CMS pneumonia model based on administrative claims data—in all aspects of model performance (discrimination, calibration, and reclassification). We found that socioeconomic status, prior hospitalizations, thrombocytosis, and measures of clinical severity and stability were important predictors of 30-day all-cause readmissions among patients hospitalized with pneumonia. Additionally, an updated discharge PSI score was a stronger independent predictor of readmissions compared to the PSI score calculated upon admission; and inclusion of the updated PSI in our full-stay pneumonia model led to improved prediction of 30-day readmissions.

The marked improvement in performance of the full-stay pneumonia-specific EHR model compared to the first-day pneumonia-specific model suggests that clinical stability and trajectory during hospitalization (as modeled through disposition status, updated PSI, and vital sign instabilities at discharge) are important predictors of 30-day readmission among patients hospitalized for pneumonia, which was not the case for our EHR-based multi-condition models.¹⁹ With the inclusion of these measures, the full-stay pneumonia-specific model correctly reclassified an additional 8% of patients according to their true risk compared to the first-day pneumonia-specific model. One implication of these findings is that hospitals interested in targeting their highest risk individuals with pneumonia for transitional care interventions could do so using the first-day pneumonia-specific EHR model and could refine their targeted strategy at the time of discharge by using the full-stay pneumonia model. This staged risk-prediction strategy would enable hospitals to initiate transitional care interventions for high-risk individuals in the inpatient setting (ie, patient education).⁷ Then, hospitals could enroll both persistent and newly identified high-risk individuals for outpatient interventions (ie, follow-up telephone call) in the immediate post-discharge period, an interval characterized by heightened vulnerability for adverse events,²⁸ based on patients’ illness severity and stability at discharge. This approach can be implemented by hospitals by building these risk-prediction models directly into the EHR, or by extracting EHR data in near real time as our group has done successfully for heart failure.⁷

Another key implication of our study is that, for pneumonia, a disease-specific modeling approach has better predictive ability than using a multi-condition model. Compared to multi-condition models, the first-day and full-stay pneumonia-specific EHR models correctly reclassified an additional 6% and 9% of patients, respectively. Thus, hospitals interested in identifying the highest risk patients with pneumonia for targeted interventions should do so using the disease-specific models, if the costs and resources of doing so are within reach of the healthcare system.

An additional novel finding of our study is the added value of an updated PSI for predicting adverse events. Studies of pneumonia severity of illness scores have calculated the PSI and CURB-65 scores using data present only on admission.^16,24 While our study also confirms that the PSI calculated upon admission is a significant predictor of readmission,^23,29 this study extends this work by showing that an updated PSI score calculated at the time of discharge is an even stronger predictor for readmission, and its inclusion in the model significantly improves risk stratification and prognostication.

Our study was noteworthy for several strengths. First, we used data from a common EHR system, thus potentially allowing for the implementation of the pneumonia-specific models in real time across a number of hospitals. The use of routinely collected data for risk-prediction modeling makes this approach scalable and sustainable, because it obviates the need for burdensome data collection and entry. Second, to our knowledge, this is the first study to measure the additive influence of illness severity and stability at discharge on the readmission risk among patients hospitalized with pneumonia. Third, our study population was derived from 6 hospitals diverse in payer status, age, race/ethnicity, and socioeconomic status. Fourth, our models are less likely to be overfit to the idiosyncrasies of our data given that several predictors included in our final pneumonia-specific models have been associated with readmission in this population, including marital status,^13,30 income,^11,31 prior hospitalizations,^11,13 thrombocytosis,^32-34 and vital sign instabilities on discharge.¹⁷ Lastly, the discrimination of the CMS pneumonia model in our cohort (C statistic, 0.64) closely matched the discrimination observed in 4 independent cohorts (C statistic, 0.63), suggesting adequate generalizability of our study setting and population.^10,12

Our results should be interpreted in the context of several limitations. First, generalizability to other regions beyond north Texas is unknown. Second, although we included a diverse cohort of safety net, community, teaching, and nonteaching hospitals, the pneumonia-specific models were not externally validated in a separate cohort, which may lead to more optimistic estimates of model performance. Third, PSI and CURB-65 scores were modified to use diagnostic codes for altered mental status and pleural effusion, and omitted nursing home residence. Thus, the independent associations for the PSI and CURB-65 scores and their predictive ability are likely attenuated. Fourth, we were unable to include data on medications (antibiotics and steroid use) and outpatient visits, which may influence readmission risk.^2,9,13,35-40 Fifth, we included only the first pneumonia hospitalization per patient in this study. Had we included multiple hospitalizations per patient, we anticipate better model performance for the 2 pneumonia-specific EHR models since prior hospitalization was a robust predictor of readmission.

In conclusion, the full-stay pneumonia-specific EHR readmission risk-prediction model outperformed the first-day pneumonia-specific model, multi-condition EHR models, and the CMS pneumonia model. This suggests that: measures of clinical severity and stability at the time of discharge are important predictors for identifying patients at highest risk for readmission; and that EHR data routinely collected for clinical practice can be used to accurately predict risk of readmission among patients hospitalized for pneumonia.

Acknowledgments

The authors would like to acknowledge Ruben Amarasingham, MD, MBA, president and chief executive officer of Parkland Center for Clinical Innovation, and Ferdinand Velasco, MD, chief health information officer at Texas Health Resources for their assistance in assembling the 6-hospital cohort used in this study.

Disclosures

This work was supported by the Agency for Healthcare Research and Quality-funded UT Southwestern Center for Patient-Centered Outcomes Research (R24 HS022418-01); the Commonwealth Foundation (#20100323); the UT Southwestern KL2 Scholars Program supported by the National Institutes of Health (KL2 TR001103 to ANM and OKN); and the National Center for Advancing Translational Sciences at the National Institute of Health (U54 RFA-TR-12-006 to E.A.H.). The study sponsors had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no financial conflicts of interest to disclose

Pneumonia is a leading cause of hospitalizations in the U.S., accounting for more than 1.1 million discharges annually.¹ Pneumonia is frequently complicated by hospital readmission, which is costly and potentially avoidable.^2,3 Due to financial penalties imposed on hospitals for higher than expected 30-day readmission rates, there is increasing attention to implementing interventions to reduce readmissions in this population.^4,5 However, because these programs are resource-intensive, interventions are thought to be most cost-effective if they are targeted to high-risk individuals who are most likely to benefit.^6-8

Current pneumonia-specific readmission risk-prediction models that could enable identification of high-risk patients suffer from poor predictive ability, greatly limiting their use, and most were validated among older adults or by using data from single academic medical centers, limiting their generalizability.^9-14 A potential reason for poor predictive accuracy is the omission of known robust clinical predictors of pneumonia-related outcomes, including pneumonia severity of illness and stability on discharge.^15-17 Approaches using electronic health record (EHR) data, which include this clinically granular data, could enable hospitals to more accurately and pragmatically identify high-risk patients during the index hospitalization and enable interventions to be initiated prior to discharge.

An alternative strategy to identifying high-risk patients for readmission is to use a multi-condition risk-prediction model. Developing and implementing models for every condition may be time-consuming and costly. We have derived and validated 2 multi-condition risk-prediction models using EHR data—1 using data from the first day of hospital admission (‘first-day’ model), and the second incorporating data from the entire hospitalization (‘full-stay’ model) to reflect in-hospital complications and clinical stability at discharge.^18,19 However, it is unknown if a multi-condition model for pneumonia would perform as well as a disease-specific model.

This study aimed to develop 2 EHR-based pneumonia-specific readmission risk-prediction models using data routinely collected in clinical practice—a ‘first-day’ and a ‘full-stay’ model—and compare the performance of each model to: 1) one another; 2) the corresponding multi-condition EHR model; and 3) to other potentially useful models in predicting pneumonia readmissions (the Centers for Medicare and Medicaid Services [CMS] pneumonia model, and 2 commonly used pneumonia severity of illness scores validated for predicting mortality). We hypothesized that the pneumonia-specific EHR models would outperform other models; and the full-stay pneumonia-specific model would outperform the first-day pneumonia-specific model.

Study Design, Population, and Data Sources

We conducted an observational study using EHR data collected from 6 hospitals (including safety net, community, teaching, and nonteaching hospitals) in north Texas between November 2009 and October 2010, All hospitals used the Epic EHR (Epic Systems Corporation, Verona, WI). Details of this cohort have been published.^18,19

We included consecutive hospitalizations among adults 18 years and older discharged from any medicine service with principal discharge diagnoses of pneumonia (ICD-9-CM codes 480-483, 485, 486-487), sepsis (ICD-9-CM codes 038, 995.91, 995.92, 785.52), or respiratory failure (ICD-9-CM codes 518.81, 518.82, 518.84, 799.1) when the latter 2 were also accompanied by a secondary diagnosis of pneumonia.²⁰ For individuals with multiple hospitalizations during the study period, we included only the first hospitalization. We excluded individuals who died during the index hospitalization or within 30 days of discharge, were transferred to another acute care facility, or left against medical advice.

Outcomes

The primary outcome was all-cause 30-day readmission, defined as a nonelective hospitalization within 30 days of discharge to any of 75 acute care hospitals within a 100-mile radius of Dallas, ascertained from an all-payer regional hospitalization database.

Predictor Variables for the Pneumonia-Specific Readmission Models

The selection of candidate predictors was informed by our validated multi-condition risk-prediction models using EHR data available within 24 hours of admission (‘first-day’ multi-condition EHR model) or during the entire hospitalization (‘full-stay’ multi-condition EHR model).^18,19 For the pneumonia-specific models, we included all variables in our published multi-condition models as candidate predictors, including sociodemographics, prior utilization, Charlson Comorbidity Index, select laboratory and vital sign abnormalities, length of stay, hospital complications (eg, venous thromboembolism), vital sign instabilities, and disposition status (see Supplemental Table 1 for complete list of variables). We also assessed additional variables specific to pneumonia for inclusion that were: (1) available in the EHR of all participating hospitals; (2) routinely collected or available at the time of admission or discharge; and (3) plausible predictors of adverse outcomes based on literature and clinical expertise. These included select comorbidities (eg, psychiatric conditions, chronic lung disease, history of pneumonia),^10,11,21,22 the pneumonia severity index (PSI),^16,23,24 intensive care unit stay, and receipt of invasive or noninvasive ventilation. We used a modified PSI score because certain data elements were missing. The modified PSI (henceforth referred to as PSI) did not include nursing home residence and included diagnostic codes as proxies for the presence of pleural effusion (ICD-9-CM codes 510, 511.1, and 511.9) and altered mental status (ICD-9-CM codes 780.0X, 780.97, 293.0, 293.1, and 348.3X).

Statistical Analysis

Model Derivation. Candidate predictor variables were classified as available in the EHR within 24 hours of admission and/or at the time of discharge. For example, socioeconomic factors could be ascertained within the first day of hospitalization, whereas length of stay would not be available until the day of discharge. Predictors with missing values were assumed to be normal (less than 1% missing for each variable). Univariate relationships between readmission and each candidate predictor were assessed in the overall cohort using a pre-specified significance threshold of P ≤ 0.10. Significant variables were entered in the respective first-day and full-stay pneumonia-specific multivariable logistic regression models using stepwise-backward selection with a pre-specified significance threshold of P ≤ 0.05. In sensitivity analyses, we alternately derived our models using stepwise-forward selection, as well as stepwise-backward selection minimizing the Bayesian information criterion and Akaike information criterion separately. These alternate modeling strategies yielded identical predictors to our final models.

Model Validation. Model validation was performed using 5-fold cross-validation, with the overall cohort randomly divided into 5 equal-size subsets.²⁵ For each cycle, 4 subsets were used for training to estimate model coefficients, and the fifth subset was used for validation. This cycle was repeated 5 times with each randomly-divided subset used once as the validation set. We repeated this entire process 50 times and averaged the C statistic estimates to derive an optimism-corrected C statistic. Model calibration was assessed qualitatively by comparing predicted to observed probabilities of readmission by quintiles of predicted risk, and with the Hosmer-Lemeshow goodness-of-fit test.

Comparison to Other Models. The main comparisons of the first-day and full-stay pneumonia-specific EHR model performance were to each other and the corresponding multi-condition EHR model.^18,19 The multi-condition EHR models were separately derived and validated within the larger parent cohort from which this study cohort was derived, and outperformed the CMS all-cause model, the HOSPITAL model, and the LACE index.¹⁹ To further triangulate our findings, given the lack of other rigorously validated pneumonia-specific risk-prediction models for readmission,¹⁴ we compared the pneumonia-specific EHR models to the CMS pneumonia model derived from administrative claims data,¹⁰ and 2 commonly used risk-prediction scores for short-term mortality among patients with community-acquired pneumonia, the PSI and CURB-65 scores.¹⁶ Although derived and validated using patient-level data, the CMS model was developed to benchmark hospitals according to hospital-level readmission rates.¹⁰ The CURB-65 score in this study was also modified to include the same altered mental status diagnostic codes according to the modified PSI as a proxy for “confusion.” Both the PSI and CURB-65 scores were calculated using the most abnormal values within the first 24 hours of admission. The ‘updated’ PSI and the ‘updated’ CURB-65 were calculated using the most abnormal values within 24 hours prior to discharge, or the last known observation prior to discharge if no results were recorded within this time period. A complete list of variables for each of the comparison models are shown in Supplemental Table 1.

We assessed model performance by calculating the C statistic, integrated discrimination index, and net reclassification index (NRI) compared to our pneumonia-specific models. The integrated discrimination index is the difference in the mean predicted probability of readmission between patients who were and were not actually readmitted between 2 models, where more positive values suggest improvement in model performance compared to a reference model.²⁶ The NRI is defined as the sum of the net proportions of correctly reclassified persons with and without the event of interest.²⁷ Here, we calculated a category-based NRI to evaluate the performance of pneumonia-specific models in correctly classifying individuals with and without readmissions into the 2 highest readmission risk quintiles vs the lowest 3 risk quintiles compared to other models.²⁷ This pre-specified cutoff is relevant for hospitals interested in identifying the highest risk individuals for targeted intervention.⁷ Finally, we assessed calibration of comparator models in our cohort by comparing predicted probability to observed probability of readmission by quintiles of risk for each model. We conducted all analyses using Stata 12.1 (StataCorp, College Station, Texas). This study was approved by the University of Texas Southwestern Medical Center Institutional Review Board.

Of 1463 index hospitalizations (Supplemental Figure 1), the 30-day all-cause readmission rate was 13.6%. Individuals with a 30-day readmission had markedly different sociodemographic and clinical characteristics compared to those not readmitted (Table 1; see Supplemental Table 2 for additional clinical characteristics).

Derivation, Validation, and Performance of the Pneumonia-Specific Readmission Risk-Prediction Models

The final first-day pneumonia-specific EHR model included 7 variables, including sociodemographic characteristics; prior hospitalizations; thrombocytosis, and PSI (Table 2). The first-day pneumonia-specific model had adequate discrimination (C statistic, 0.695; optimism-corrected C statistic 0.675, 95% confidence interval [CI], 0.667-0.685; Table 3). It also effectively stratified individuals across a broad range of risk (average predicted decile of risk ranged from 4% to 33%; Table 3) and was well calibrated (Supplemental Table 3).

The final full-stay pneumonia-specific EHR readmission model included 8 predictors, including 3 variables from the first-day model (median income, thrombocytosis, and prior hospitalizations; Table 2). The full-stay pneumonia-specific EHR model also included vital sign instabilities on discharge, updated PSI, and disposition status (ie, being discharged with home health or to a post-acute care facility was associated with greater odds of readmission, and hospice with lower odds). The full-stay pneumonia-specific EHR model had good discrimination (C statistic, 0.731; optimism-corrected C statistic, 0.714; 95% CI, 0.706-0.720), and stratified individuals across a broad range of risk (average predicted decile of risk ranged from 3% to 37%; Table 3), and was also well calibrated (Supplemental Table 3).

First-Day Pneumonia-Specific EHR Model vs First-Day Multi-Condition EHR Model

The first-day pneumonia-specific EHR model outperformed the first-day multi-condition EHR model with better discrimination (P = 0.029) and more correctly classified individuals in the top 2 highest risk quintiles vs the bottom 3 risk quintiles (Table 3, Supplemental Table 4, and Supplemental Figure 2A). With respect to calibration, the first-day multi-condition EHR model overestimated risk among the highest quintile risk group compared to the first-day pneumonia-specific EHR model (Figure 1A, 1B).

Full-Stay Pneumonia-Specific EHR Model vs Other Models

The full-stay pneumonia-specific EHR model comparatively outperformed the corresponding full-stay multi-condition EHR model, as well as the first-day pneumonia-specific EHR model, the CMS pneumonia model, the updated PSI, and the updated CURB-65 (Table 3, Supplemental Table 5, Supplemental Table 6, and Supplemental Figures 2B and 2C). Compared to the full-stay multi-condition and first-day pneumonia-specific EHR models, the full-stay pneumonia-specific EHR model had better discrimination, better reclassification (NRI, 0.09 and 0.08, respectively), and was able to stratify individuals across a broader range of readmission risk (Table 3). It also had better calibration in the highest quintile risk group compared to the full-stay multi-condition EHR model (Figure 1C and 1D).

Updated vs First-Day Modified PSI and CURB-65 Scores

The updated PSI was more strongly predictive of readmission than the PSI calculated on the day of admission (Wald test, 9.83; P = 0.002). Each 10-point increase in the updated PSI was associated with a 22% increased odds of readmission vs an 11% increase for the PSI calculated upon admission (Table 2). The improved predictive ability of the updated PSI and CURB-65 scores was also reflected in the superior discrimination and calibration vs the respective first-day pneumonia severity of illness scores (Table 3).

Using routinely available EHR data from 6 diverse hospitals, we developed 2 pneumonia-specific readmission risk-prediction models that aimed to allow hospitals to identify patients hospitalized with pneumonia at high risk for readmission. Overall, we found that a pneumonia-specific model using EHR data from the entire hospitalization outperformed all other models—including the first-day pneumonia-specific model using data present only on admission, our own multi-condition EHR models, and the CMS pneumonia model based on administrative claims data—in all aspects of model performance (discrimination, calibration, and reclassification). We found that socioeconomic status, prior hospitalizations, thrombocytosis, and measures of clinical severity and stability were important predictors of 30-day all-cause readmissions among patients hospitalized with pneumonia. Additionally, an updated discharge PSI score was a stronger independent predictor of readmissions compared to the PSI score calculated upon admission; and inclusion of the updated PSI in our full-stay pneumonia model led to improved prediction of 30-day readmissions.

The marked improvement in performance of the full-stay pneumonia-specific EHR model compared to the first-day pneumonia-specific model suggests that clinical stability and trajectory during hospitalization (as modeled through disposition status, updated PSI, and vital sign instabilities at discharge) are important predictors of 30-day readmission among patients hospitalized for pneumonia, which was not the case for our EHR-based multi-condition models.¹⁹ With the inclusion of these measures, the full-stay pneumonia-specific model correctly reclassified an additional 8% of patients according to their true risk compared to the first-day pneumonia-specific model. One implication of these findings is that hospitals interested in targeting their highest risk individuals with pneumonia for transitional care interventions could do so using the first-day pneumonia-specific EHR model and could refine their targeted strategy at the time of discharge by using the full-stay pneumonia model. This staged risk-prediction strategy would enable hospitals to initiate transitional care interventions for high-risk individuals in the inpatient setting (ie, patient education).⁷ Then, hospitals could enroll both persistent and newly identified high-risk individuals for outpatient interventions (ie, follow-up telephone call) in the immediate post-discharge period, an interval characterized by heightened vulnerability for adverse events,²⁸ based on patients’ illness severity and stability at discharge. This approach can be implemented by hospitals by building these risk-prediction models directly into the EHR, or by extracting EHR data in near real time as our group has done successfully for heart failure.⁷

Another key implication of our study is that, for pneumonia, a disease-specific modeling approach has better predictive ability than using a multi-condition model. Compared to multi-condition models, the first-day and full-stay pneumonia-specific EHR models correctly reclassified an additional 6% and 9% of patients, respectively. Thus, hospitals interested in identifying the highest risk patients with pneumonia for targeted interventions should do so using the disease-specific models, if the costs and resources of doing so are within reach of the healthcare system.

An additional novel finding of our study is the added value of an updated PSI for predicting adverse events. Studies of pneumonia severity of illness scores have calculated the PSI and CURB-65 scores using data present only on admission.^16,24 While our study also confirms that the PSI calculated upon admission is a significant predictor of readmission,^23,29 this study extends this work by showing that an updated PSI score calculated at the time of discharge is an even stronger predictor for readmission, and its inclusion in the model significantly improves risk stratification and prognostication.

Our study was noteworthy for several strengths. First, we used data from a common EHR system, thus potentially allowing for the implementation of the pneumonia-specific models in real time across a number of hospitals. The use of routinely collected data for risk-prediction modeling makes this approach scalable and sustainable, because it obviates the need for burdensome data collection and entry. Second, to our knowledge, this is the first study to measure the additive influence of illness severity and stability at discharge on the readmission risk among patients hospitalized with pneumonia. Third, our study population was derived from 6 hospitals diverse in payer status, age, race/ethnicity, and socioeconomic status. Fourth, our models are less likely to be overfit to the idiosyncrasies of our data given that several predictors included in our final pneumonia-specific models have been associated with readmission in this population, including marital status,^13,30 income,^11,31 prior hospitalizations,^11,13 thrombocytosis,^32-34 and vital sign instabilities on discharge.¹⁷ Lastly, the discrimination of the CMS pneumonia model in our cohort (C statistic, 0.64) closely matched the discrimination observed in 4 independent cohorts (C statistic, 0.63), suggesting adequate generalizability of our study setting and population.^10,12

Our results should be interpreted in the context of several limitations. First, generalizability to other regions beyond north Texas is unknown. Second, although we included a diverse cohort of safety net, community, teaching, and nonteaching hospitals, the pneumonia-specific models were not externally validated in a separate cohort, which may lead to more optimistic estimates of model performance. Third, PSI and CURB-65 scores were modified to use diagnostic codes for altered mental status and pleural effusion, and omitted nursing home residence. Thus, the independent associations for the PSI and CURB-65 scores and their predictive ability are likely attenuated. Fourth, we were unable to include data on medications (antibiotics and steroid use) and outpatient visits, which may influence readmission risk.^2,9,13,35-40 Fifth, we included only the first pneumonia hospitalization per patient in this study. Had we included multiple hospitalizations per patient, we anticipate better model performance for the 2 pneumonia-specific EHR models since prior hospitalization was a robust predictor of readmission.

In conclusion, the full-stay pneumonia-specific EHR readmission risk-prediction model outperformed the first-day pneumonia-specific model, multi-condition EHR models, and the CMS pneumonia model. This suggests that: measures of clinical severity and stability at the time of discharge are important predictors for identifying patients at highest risk for readmission; and that EHR data routinely collected for clinical practice can be used to accurately predict risk of readmission among patients hospitalized for pneumonia.

Acknowledgments

The authors would like to acknowledge Ruben Amarasingham, MD, MBA, president and chief executive officer of Parkland Center for Clinical Innovation, and Ferdinand Velasco, MD, chief health information officer at Texas Health Resources for their assistance in assembling the 6-hospital cohort used in this study.

Disclosures

This work was supported by the Agency for Healthcare Research and Quality-funded UT Southwestern Center for Patient-Centered Outcomes Research (R24 HS022418-01); the Commonwealth Foundation (#20100323); the UT Southwestern KL2 Scholars Program supported by the National Institutes of Health (KL2 TR001103 to ANM and OKN); and the National Center for Advancing Translational Sciences at the National Institute of Health (U54 RFA-TR-12-006 to E.A.H.). The study sponsors had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no financial conflicts of interest to disclose

Pneumonia is a leading cause of hospitalizations in the U.S., accounting for more than 1.1 million discharges annually.¹ Pneumonia is frequently complicated by hospital readmission, which is costly and potentially avoidable.^2,3 Due to financial penalties imposed on hospitals for higher than expected 30-day readmission rates, there is increasing attention to implementing interventions to reduce readmissions in this population.^4,5 However, because these programs are resource-intensive, interventions are thought to be most cost-effective if they are targeted to high-risk individuals who are most likely to benefit.^6-8

Current pneumonia-specific readmission risk-prediction models that could enable identification of high-risk patients suffer from poor predictive ability, greatly limiting their use, and most were validated among older adults or by using data from single academic medical centers, limiting their generalizability.^9-14 A potential reason for poor predictive accuracy is the omission of known robust clinical predictors of pneumonia-related outcomes, including pneumonia severity of illness and stability on discharge.^15-17 Approaches using electronic health record (EHR) data, which include this clinically granular data, could enable hospitals to more accurately and pragmatically identify high-risk patients during the index hospitalization and enable interventions to be initiated prior to discharge.

An alternative strategy to identifying high-risk patients for readmission is to use a multi-condition risk-prediction model. Developing and implementing models for every condition may be time-consuming and costly. We have derived and validated 2 multi-condition risk-prediction models using EHR data—1 using data from the first day of hospital admission (‘first-day’ model), and the second incorporating data from the entire hospitalization (‘full-stay’ model) to reflect in-hospital complications and clinical stability at discharge.^18,19 However, it is unknown if a multi-condition model for pneumonia would perform as well as a disease-specific model.

This study aimed to develop 2 EHR-based pneumonia-specific readmission risk-prediction models using data routinely collected in clinical practice—a ‘first-day’ and a ‘full-stay’ model—and compare the performance of each model to: 1) one another; 2) the corresponding multi-condition EHR model; and 3) to other potentially useful models in predicting pneumonia readmissions (the Centers for Medicare and Medicaid Services [CMS] pneumonia model, and 2 commonly used pneumonia severity of illness scores validated for predicting mortality). We hypothesized that the pneumonia-specific EHR models would outperform other models; and the full-stay pneumonia-specific model would outperform the first-day pneumonia-specific model.

Study Design, Population, and Data Sources

We conducted an observational study using EHR data collected from 6 hospitals (including safety net, community, teaching, and nonteaching hospitals) in north Texas between November 2009 and October 2010, All hospitals used the Epic EHR (Epic Systems Corporation, Verona, WI). Details of this cohort have been published.^18,19

We included consecutive hospitalizations among adults 18 years and older discharged from any medicine service with principal discharge diagnoses of pneumonia (ICD-9-CM codes 480-483, 485, 486-487), sepsis (ICD-9-CM codes 038, 995.91, 995.92, 785.52), or respiratory failure (ICD-9-CM codes 518.81, 518.82, 518.84, 799.1) when the latter 2 were also accompanied by a secondary diagnosis of pneumonia.²⁰ For individuals with multiple hospitalizations during the study period, we included only the first hospitalization. We excluded individuals who died during the index hospitalization or within 30 days of discharge, were transferred to another acute care facility, or left against medical advice.

Outcomes

The primary outcome was all-cause 30-day readmission, defined as a nonelective hospitalization within 30 days of discharge to any of 75 acute care hospitals within a 100-mile radius of Dallas, ascertained from an all-payer regional hospitalization database.

Predictor Variables for the Pneumonia-Specific Readmission Models

The selection of candidate predictors was informed by our validated multi-condition risk-prediction models using EHR data available within 24 hours of admission (‘first-day’ multi-condition EHR model) or during the entire hospitalization (‘full-stay’ multi-condition EHR model).^18,19 For the pneumonia-specific models, we included all variables in our published multi-condition models as candidate predictors, including sociodemographics, prior utilization, Charlson Comorbidity Index, select laboratory and vital sign abnormalities, length of stay, hospital complications (eg, venous thromboembolism), vital sign instabilities, and disposition status (see Supplemental Table 1 for complete list of variables). We also assessed additional variables specific to pneumonia for inclusion that were: (1) available in the EHR of all participating hospitals; (2) routinely collected or available at the time of admission or discharge; and (3) plausible predictors of adverse outcomes based on literature and clinical expertise. These included select comorbidities (eg, psychiatric conditions, chronic lung disease, history of pneumonia),^10,11,21,22 the pneumonia severity index (PSI),^16,23,24 intensive care unit stay, and receipt of invasive or noninvasive ventilation. We used a modified PSI score because certain data elements were missing. The modified PSI (henceforth referred to as PSI) did not include nursing home residence and included diagnostic codes as proxies for the presence of pleural effusion (ICD-9-CM codes 510, 511.1, and 511.9) and altered mental status (ICD-9-CM codes 780.0X, 780.97, 293.0, 293.1, and 348.3X).

Statistical Analysis

Model Derivation. Candidate predictor variables were classified as available in the EHR within 24 hours of admission and/or at the time of discharge. For example, socioeconomic factors could be ascertained within the first day of hospitalization, whereas length of stay would not be available until the day of discharge. Predictors with missing values were assumed to be normal (less than 1% missing for each variable). Univariate relationships between readmission and each candidate predictor were assessed in the overall cohort using a pre-specified significance threshold of P ≤ 0.10. Significant variables were entered in the respective first-day and full-stay pneumonia-specific multivariable logistic regression models using stepwise-backward selection with a pre-specified significance threshold of P ≤ 0.05. In sensitivity analyses, we alternately derived our models using stepwise-forward selection, as well as stepwise-backward selection minimizing the Bayesian information criterion and Akaike information criterion separately. These alternate modeling strategies yielded identical predictors to our final models.

Model Validation. Model validation was performed using 5-fold cross-validation, with the overall cohort randomly divided into 5 equal-size subsets.²⁵ For each cycle, 4 subsets were used for training to estimate model coefficients, and the fifth subset was used for validation. This cycle was repeated 5 times with each randomly-divided subset used once as the validation set. We repeated this entire process 50 times and averaged the C statistic estimates to derive an optimism-corrected C statistic. Model calibration was assessed qualitatively by comparing predicted to observed probabilities of readmission by quintiles of predicted risk, and with the Hosmer-Lemeshow goodness-of-fit test.

Comparison to Other Models. The main comparisons of the first-day and full-stay pneumonia-specific EHR model performance were to each other and the corresponding multi-condition EHR model.^18,19 The multi-condition EHR models were separately derived and validated within the larger parent cohort from which this study cohort was derived, and outperformed the CMS all-cause model, the HOSPITAL model, and the LACE index.¹⁹ To further triangulate our findings, given the lack of other rigorously validated pneumonia-specific risk-prediction models for readmission,¹⁴ we compared the pneumonia-specific EHR models to the CMS pneumonia model derived from administrative claims data,¹⁰ and 2 commonly used risk-prediction scores for short-term mortality among patients with community-acquired pneumonia, the PSI and CURB-65 scores.¹⁶ Although derived and validated using patient-level data, the CMS model was developed to benchmark hospitals according to hospital-level readmission rates.¹⁰ The CURB-65 score in this study was also modified to include the same altered mental status diagnostic codes according to the modified PSI as a proxy for “confusion.” Both the PSI and CURB-65 scores were calculated using the most abnormal values within the first 24 hours of admission. The ‘updated’ PSI and the ‘updated’ CURB-65 were calculated using the most abnormal values within 24 hours prior to discharge, or the last known observation prior to discharge if no results were recorded within this time period. A complete list of variables for each of the comparison models are shown in Supplemental Table 1.

We assessed model performance by calculating the C statistic, integrated discrimination index, and net reclassification index (NRI) compared to our pneumonia-specific models. The integrated discrimination index is the difference in the mean predicted probability of readmission between patients who were and were not actually readmitted between 2 models, where more positive values suggest improvement in model performance compared to a reference model.²⁶ The NRI is defined as the sum of the net proportions of correctly reclassified persons with and without the event of interest.²⁷ Here, we calculated a category-based NRI to evaluate the performance of pneumonia-specific models in correctly classifying individuals with and without readmissions into the 2 highest readmission risk quintiles vs the lowest 3 risk quintiles compared to other models.²⁷ This pre-specified cutoff is relevant for hospitals interested in identifying the highest risk individuals for targeted intervention.⁷ Finally, we assessed calibration of comparator models in our cohort by comparing predicted probability to observed probability of readmission by quintiles of risk for each model. We conducted all analyses using Stata 12.1 (StataCorp, College Station, Texas). This study was approved by the University of Texas Southwestern Medical Center Institutional Review Board.

Of 1463 index hospitalizations (Supplemental Figure 1), the 30-day all-cause readmission rate was 13.6%. Individuals with a 30-day readmission had markedly different sociodemographic and clinical characteristics compared to those not readmitted (Table 1; see Supplemental Table 2 for additional clinical characteristics).

Derivation, Validation, and Performance of the Pneumonia-Specific Readmission Risk-Prediction Models

The final first-day pneumonia-specific EHR model included 7 variables, including sociodemographic characteristics; prior hospitalizations; thrombocytosis, and PSI (Table 2). The first-day pneumonia-specific model had adequate discrimination (C statistic, 0.695; optimism-corrected C statistic 0.675, 95% confidence interval [CI], 0.667-0.685; Table 3). It also effectively stratified individuals across a broad range of risk (average predicted decile of risk ranged from 4% to 33%; Table 3) and was well calibrated (Supplemental Table 3).

The final full-stay pneumonia-specific EHR readmission model included 8 predictors, including 3 variables from the first-day model (median income, thrombocytosis, and prior hospitalizations; Table 2). The full-stay pneumonia-specific EHR model also included vital sign instabilities on discharge, updated PSI, and disposition status (ie, being discharged with home health or to a post-acute care facility was associated with greater odds of readmission, and hospice with lower odds). The full-stay pneumonia-specific EHR model had good discrimination (C statistic, 0.731; optimism-corrected C statistic, 0.714; 95% CI, 0.706-0.720), and stratified individuals across a broad range of risk (average predicted decile of risk ranged from 3% to 37%; Table 3), and was also well calibrated (Supplemental Table 3).

First-Day Pneumonia-Specific EHR Model vs First-Day Multi-Condition EHR Model

The first-day pneumonia-specific EHR model outperformed the first-day multi-condition EHR model with better discrimination (P = 0.029) and more correctly classified individuals in the top 2 highest risk quintiles vs the bottom 3 risk quintiles (Table 3, Supplemental Table 4, and Supplemental Figure 2A). With respect to calibration, the first-day multi-condition EHR model overestimated risk among the highest quintile risk group compared to the first-day pneumonia-specific EHR model (Figure 1A, 1B).

Full-Stay Pneumonia-Specific EHR Model vs Other Models

The full-stay pneumonia-specific EHR model comparatively outperformed the corresponding full-stay multi-condition EHR model, as well as the first-day pneumonia-specific EHR model, the CMS pneumonia model, the updated PSI, and the updated CURB-65 (Table 3, Supplemental Table 5, Supplemental Table 6, and Supplemental Figures 2B and 2C). Compared to the full-stay multi-condition and first-day pneumonia-specific EHR models, the full-stay pneumonia-specific EHR model had better discrimination, better reclassification (NRI, 0.09 and 0.08, respectively), and was able to stratify individuals across a broader range of readmission risk (Table 3). It also had better calibration in the highest quintile risk group compared to the full-stay multi-condition EHR model (Figure 1C and 1D).

Updated vs First-Day Modified PSI and CURB-65 Scores

The updated PSI was more strongly predictive of readmission than the PSI calculated on the day of admission (Wald test, 9.83; P = 0.002). Each 10-point increase in the updated PSI was associated with a 22% increased odds of readmission vs an 11% increase for the PSI calculated upon admission (Table 2). The improved predictive ability of the updated PSI and CURB-65 scores was also reflected in the superior discrimination and calibration vs the respective first-day pneumonia severity of illness scores (Table 3).

Using routinely available EHR data from 6 diverse hospitals, we developed 2 pneumonia-specific readmission risk-prediction models that aimed to allow hospitals to identify patients hospitalized with pneumonia at high risk for readmission. Overall, we found that a pneumonia-specific model using EHR data from the entire hospitalization outperformed all other models—including the first-day pneumonia-specific model using data present only on admission, our own multi-condition EHR models, and the CMS pneumonia model based on administrative claims data—in all aspects of model performance (discrimination, calibration, and reclassification). We found that socioeconomic status, prior hospitalizations, thrombocytosis, and measures of clinical severity and stability were important predictors of 30-day all-cause readmissions among patients hospitalized with pneumonia. Additionally, an updated discharge PSI score was a stronger independent predictor of readmissions compared to the PSI score calculated upon admission; and inclusion of the updated PSI in our full-stay pneumonia model led to improved prediction of 30-day readmissions.

The marked improvement in performance of the full-stay pneumonia-specific EHR model compared to the first-day pneumonia-specific model suggests that clinical stability and trajectory during hospitalization (as modeled through disposition status, updated PSI, and vital sign instabilities at discharge) are important predictors of 30-day readmission among patients hospitalized for pneumonia, which was not the case for our EHR-based multi-condition models.¹⁹ With the inclusion of these measures, the full-stay pneumonia-specific model correctly reclassified an additional 8% of patients according to their true risk compared to the first-day pneumonia-specific model. One implication of these findings is that hospitals interested in targeting their highest risk individuals with pneumonia for transitional care interventions could do so using the first-day pneumonia-specific EHR model and could refine their targeted strategy at the time of discharge by using the full-stay pneumonia model. This staged risk-prediction strategy would enable hospitals to initiate transitional care interventions for high-risk individuals in the inpatient setting (ie, patient education).⁷ Then, hospitals could enroll both persistent and newly identified high-risk individuals for outpatient interventions (ie, follow-up telephone call) in the immediate post-discharge period, an interval characterized by heightened vulnerability for adverse events,²⁸ based on patients’ illness severity and stability at discharge. This approach can be implemented by hospitals by building these risk-prediction models directly into the EHR, or by extracting EHR data in near real time as our group has done successfully for heart failure.⁷

Another key implication of our study is that, for pneumonia, a disease-specific modeling approach has better predictive ability than using a multi-condition model. Compared to multi-condition models, the first-day and full-stay pneumonia-specific EHR models correctly reclassified an additional 6% and 9% of patients, respectively. Thus, hospitals interested in identifying the highest risk patients with pneumonia for targeted interventions should do so using the disease-specific models, if the costs and resources of doing so are within reach of the healthcare system.

An additional novel finding of our study is the added value of an updated PSI for predicting adverse events. Studies of pneumonia severity of illness scores have calculated the PSI and CURB-65 scores using data present only on admission.^16,24 While our study also confirms that the PSI calculated upon admission is a significant predictor of readmission,^23,29 this study extends this work by showing that an updated PSI score calculated at the time of discharge is an even stronger predictor for readmission, and its inclusion in the model significantly improves risk stratification and prognostication.

Our study was noteworthy for several strengths. First, we used data from a common EHR system, thus potentially allowing for the implementation of the pneumonia-specific models in real time across a number of hospitals. The use of routinely collected data for risk-prediction modeling makes this approach scalable and sustainable, because it obviates the need for burdensome data collection and entry. Second, to our knowledge, this is the first study to measure the additive influence of illness severity and stability at discharge on the readmission risk among patients hospitalized with pneumonia. Third, our study population was derived from 6 hospitals diverse in payer status, age, race/ethnicity, and socioeconomic status. Fourth, our models are less likely to be overfit to the idiosyncrasies of our data given that several predictors included in our final pneumonia-specific models have been associated with readmission in this population, including marital status,^13,30 income,^11,31 prior hospitalizations,^11,13 thrombocytosis,^32-34 and vital sign instabilities on discharge.¹⁷ Lastly, the discrimination of the CMS pneumonia model in our cohort (C statistic, 0.64) closely matched the discrimination observed in 4 independent cohorts (C statistic, 0.63), suggesting adequate generalizability of our study setting and population.^10,12

Our results should be interpreted in the context of several limitations. First, generalizability to other regions beyond north Texas is unknown. Second, although we included a diverse cohort of safety net, community, teaching, and nonteaching hospitals, the pneumonia-specific models were not externally validated in a separate cohort, which may lead to more optimistic estimates of model performance. Third, PSI and CURB-65 scores were modified to use diagnostic codes for altered mental status and pleural effusion, and omitted nursing home residence. Thus, the independent associations for the PSI and CURB-65 scores and their predictive ability are likely attenuated. Fourth, we were unable to include data on medications (antibiotics and steroid use) and outpatient visits, which may influence readmission risk.^2,9,13,35-40 Fifth, we included only the first pneumonia hospitalization per patient in this study. Had we included multiple hospitalizations per patient, we anticipate better model performance for the 2 pneumonia-specific EHR models since prior hospitalization was a robust predictor of readmission.

In conclusion, the full-stay pneumonia-specific EHR readmission risk-prediction model outperformed the first-day pneumonia-specific model, multi-condition EHR models, and the CMS pneumonia model. This suggests that: measures of clinical severity and stability at the time of discharge are important predictors for identifying patients at highest risk for readmission; and that EHR data routinely collected for clinical practice can be used to accurately predict risk of readmission among patients hospitalized for pneumonia.

Acknowledgments

The authors would like to acknowledge Ruben Amarasingham, MD, MBA, president and chief executive officer of Parkland Center for Clinical Innovation, and Ferdinand Velasco, MD, chief health information officer at Texas Health Resources for their assistance in assembling the 6-hospital cohort used in this study.

Disclosures

This work was supported by the Agency for Healthcare Research and Quality-funded UT Southwestern Center for Patient-Centered Outcomes Research (R24 HS022418-01); the Commonwealth Foundation (#20100323); the UT Southwestern KL2 Scholars Program supported by the National Institutes of Health (KL2 TR001103 to ANM and OKN); and the National Center for Advancing Translational Sciences at the National Institute of Health (U54 RFA-TR-12-006 to E.A.H.). The study sponsors had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no financial conflicts of interest to disclose

BACKGROUND: Readmissions after hospitalization for pneumonia are common, but the few risk-prediction models have poor to modest predictive ability. Data routinely collected in the EHR may improve prediction.

RESULTS: Of 1,463 patients, 13.6% were readmitted. The first-day, pneumonia-specific model included sociodemographic factors, prior hospitalizations, thrombocytosis, and a modified pneumonia severity index. The full-stay model included disposition status, vital sign instabilities on discharge, and an updated pneumonia severity index calculated using values from the day of discharge as additional predictors. The full-stay, pneumonia-specific model outperformed the first-day model (C-statistic, 0.731 vs. 0.695; P = .02; net reclassification index = 0.08). Compared with a validated multicondition readmission model, the Centers for Medicare & Medicaid Services pneumonia model, and two commonly used pneumonia severity of illness scores, the full-stay pneumonia-specific model had better discrimination (C-statistic, 0.604-0.681; P less than 0.01 for all comparisons), predicted a broader range of risk, and better reclassified individuals by their true risk (net reclassification index range, 0.09-0.18).

CONCLUSIONS: EHR data collected from the entire hospitalization can accurately predict readmission risk among patients hospitalized for pneumonia. This approach outperforms a first-day, pneumonia-specific model, the Centers for Medicare & Medicaid Services pneumonia model, and two commonly used pneumonia severity of illness scores.
 

Also In JHM This Month

Evaluating automated rules for rapid response system alarm triggers in medical and surgical patients
AUTHORS: Santiago Romero-Brufau, MD; Bruce W. Morlan, MS; Matthew Johnson, MPH; Joel Hickman; Lisa L. Kirkland, MD; James M. Naessens, ScD; Jeanne Huddleston, MD, FACP, FHM

Prognosticating with the Hospital-Patient One-year Mortality Risk score using information abstracted from the medical record
AUTHORS: Genevieve Casey, MD, and Carl van Walraven, MD, FRCPC, MSc

Automating venous thromboembolism risk calculation using electronic health record data upon hospital admission: The Automated Padua Prediction Score
AUTHORS: Pierre Elias, MD; Raman Khanna, MD; Adams Dudley, MD, MBA; Jason Davies, MD, PhD; Ronald Jacolbia, MSN; Kara McArthur, BA; Andrew D. Auerbach, MD, MPH, SFHM

The value of ultrasound in cellulitis to rule out deep venous thrombosis
AUTHORS: Hyung J. Cho, MD, and Andrew S. Dunn, MD, SFHM

Hospital medicine and perioperative care: A framework for high quality, high value collaborative care
AUTHORS: Rachel E. Thompson, MD, MPH, SFHM; Kurt Pfeifer, MD, FHM; Paul Grant, MD, SFHM; Cornelia Taylor, MD; Barbara Slawski, MD, FACP, MS, SFHM; Christopher Whinney, MD, FACP, FHM; Laurence Wellikson, MD, MHM; Amir K. Jaffer, MD, MBA, SFHM
 

BACKGROUND: Readmissions after hospitalization for pneumonia are common, but the few risk-prediction models have poor to modest predictive ability. Data routinely collected in the EHR may improve prediction.

RESULTS: Of 1,463 patients, 13.6% were readmitted. The first-day, pneumonia-specific model included sociodemographic factors, prior hospitalizations, thrombocytosis, and a modified pneumonia severity index. The full-stay model included disposition status, vital sign instabilities on discharge, and an updated pneumonia severity index calculated using values from the day of discharge as additional predictors. The full-stay, pneumonia-specific model outperformed the first-day model (C-statistic, 0.731 vs. 0.695; P = .02; net reclassification index = 0.08). Compared with a validated multicondition readmission model, the Centers for Medicare & Medicaid Services pneumonia model, and two commonly used pneumonia severity of illness scores, the full-stay pneumonia-specific model had better discrimination (C-statistic, 0.604-0.681; P less than 0.01 for all comparisons), predicted a broader range of risk, and better reclassified individuals by their true risk (net reclassification index range, 0.09-0.18).

CONCLUSIONS: EHR data collected from the entire hospitalization can accurately predict readmission risk among patients hospitalized for pneumonia. This approach outperforms a first-day, pneumonia-specific model, the Centers for Medicare & Medicaid Services pneumonia model, and two commonly used pneumonia severity of illness scores.
 

Also In JHM This Month

Evaluating automated rules for rapid response system alarm triggers in medical and surgical patients
AUTHORS: Santiago Romero-Brufau, MD; Bruce W. Morlan, MS; Matthew Johnson, MPH; Joel Hickman; Lisa L. Kirkland, MD; James M. Naessens, ScD; Jeanne Huddleston, MD, FACP, FHM

Prognosticating with the Hospital-Patient One-year Mortality Risk score using information abstracted from the medical record
AUTHORS: Genevieve Casey, MD, and Carl van Walraven, MD, FRCPC, MSc

Automating venous thromboembolism risk calculation using electronic health record data upon hospital admission: The Automated Padua Prediction Score
AUTHORS: Pierre Elias, MD; Raman Khanna, MD; Adams Dudley, MD, MBA; Jason Davies, MD, PhD; Ronald Jacolbia, MSN; Kara McArthur, BA; Andrew D. Auerbach, MD, MPH, SFHM

The value of ultrasound in cellulitis to rule out deep venous thrombosis
AUTHORS: Hyung J. Cho, MD, and Andrew S. Dunn, MD, SFHM

Hospital medicine and perioperative care: A framework for high quality, high value collaborative care
AUTHORS: Rachel E. Thompson, MD, MPH, SFHM; Kurt Pfeifer, MD, FHM; Paul Grant, MD, SFHM; Cornelia Taylor, MD; Barbara Slawski, MD, FACP, MS, SFHM; Christopher Whinney, MD, FACP, FHM; Laurence Wellikson, MD, MHM; Amir K. Jaffer, MD, MBA, SFHM
 

BACKGROUND: Readmissions after hospitalization for pneumonia are common, but the few risk-prediction models have poor to modest predictive ability. Data routinely collected in the EHR may improve prediction.

RESULTS: Of 1,463 patients, 13.6% were readmitted. The first-day, pneumonia-specific model included sociodemographic factors, prior hospitalizations, thrombocytosis, and a modified pneumonia severity index. The full-stay model included disposition status, vital sign instabilities on discharge, and an updated pneumonia severity index calculated using values from the day of discharge as additional predictors. The full-stay, pneumonia-specific model outperformed the first-day model (C-statistic, 0.731 vs. 0.695; P = .02; net reclassification index = 0.08). Compared with a validated multicondition readmission model, the Centers for Medicare & Medicaid Services pneumonia model, and two commonly used pneumonia severity of illness scores, the full-stay pneumonia-specific model had better discrimination (C-statistic, 0.604-0.681; P less than 0.01 for all comparisons), predicted a broader range of risk, and better reclassified individuals by their true risk (net reclassification index range, 0.09-0.18).

CONCLUSIONS: EHR data collected from the entire hospitalization can accurately predict readmission risk among patients hospitalized for pneumonia. This approach outperforms a first-day, pneumonia-specific model, the Centers for Medicare & Medicaid Services pneumonia model, and two commonly used pneumonia severity of illness scores.
 

Also In JHM This Month

Evaluating automated rules for rapid response system alarm triggers in medical and surgical patients
AUTHORS: Santiago Romero-Brufau, MD; Bruce W. Morlan, MS; Matthew Johnson, MPH; Joel Hickman; Lisa L. Kirkland, MD; James M. Naessens, ScD; Jeanne Huddleston, MD, FACP, FHM

Prognosticating with the Hospital-Patient One-year Mortality Risk score using information abstracted from the medical record
AUTHORS: Genevieve Casey, MD, and Carl van Walraven, MD, FRCPC, MSc

Automating venous thromboembolism risk calculation using electronic health record data upon hospital admission: The Automated Padua Prediction Score
AUTHORS: Pierre Elias, MD; Raman Khanna, MD; Adams Dudley, MD, MBA; Jason Davies, MD, PhD; Ronald Jacolbia, MSN; Kara McArthur, BA; Andrew D. Auerbach, MD, MPH, SFHM

The value of ultrasound in cellulitis to rule out deep venous thrombosis
AUTHORS: Hyung J. Cho, MD, and Andrew S. Dunn, MD, SFHM

Hospital medicine and perioperative care: A framework for high quality, high value collaborative care
AUTHORS: Rachel E. Thompson, MD, MPH, SFHM; Kurt Pfeifer, MD, FHM; Paul Grant, MD, SFHM; Cornelia Taylor, MD; Barbara Slawski, MD, FACP, MS, SFHM; Christopher Whinney, MD, FACP, FHM; Laurence Wellikson, MD, MHM; Amir K. Jaffer, MD, MBA, SFHM