John D. Greene, MA

In the landmark Best Care at Lower Cost report, the Institute of Medicine presents a compelling vision of a US healthcare system where science, information technology, incentives, and care culture are brought together seamlessly to produce high‐quality healthcare.[1] At the center of this transformation is the learning healthcare system, a system characterized by its ability to leverage data arising from care provision to drive rapid improvements in care delivery.[2] When steeped within the right organizational milieu, these data help to close the virtuous cycle of continuous learning moving from science to evidence to care and back to new science. The anticipated end result is a healthcare system that can provide Americans with superior care at lower cost.

Hospital‐based practitioners will recognize the inpatient setting as an ideal demonstration opportunity for continuous learning. Hospital care is costly, accounting for more than 30% of all US healthcare costs[3]; intensive care alone accounts for a notable proportion of the US gross domestic product.[4] Inpatient care is associated with significant mortality and morbidity, and its use is often greatly increased in patients' last days.[5, 6] Fortunately, the inpatient setting also offers an ideal opportunity to leverage high‐quality data to help inform and improve care. The digitization of medicine means that far more data are now available through electronic health records, medical devices, and tests.[7] This is particularly true for inpatients, for whom a large volume of data are produced even over relatively short hospital stays.

Whereas the challenge to improve hospital care is daunting, there is an incredible opportunity to advance the quality of inpatient care through realizing the vision of the learning hospital system. In the sections that follow, we use an object lessonsepsis care within hospitals of the Kaiser Permanente Northern California (KPNC) integrated healthcare delivery systemto evaluate the challenges and insights gleaned from working toward building a learning hospital system. Then, we describe further steps that could enhance the use of inpatient data to drive improved care.

THE FRAMEWORK OF A LEARNING HEALTHCARE SYSTEM

Best Care at Lower Cost notes a fundamental paradox in US healthcare: although we have witnessed a dramatic expansion in biomedical knowledge, innovative therapies and surgical procedures, and clinical treatments to extend survival, US healthcare persistently falls short on the basic dimensions of quality, outcomes, cost, and equity.[1] The proposed path forward lies in building the learning healthcare system, a system characterized by continuous knowledge development, improvement, and application. Figure 1 shows the critical nodes in the framework for continuous learning, which include: (1) the development of new scientific knowledge (science), (2) the translation of science into clinical evidence of efficacy (evidence), and (3) the application of efficacious interventions through effective care delivery (care). In healthcare today, transitions between these nodes are rife with missed or wasted opportunities like delays in applying high‐quality evidence or poorly managed insights arising from scientific discovery. If such opportunities could be recovered, however, the quality of healthcare could be improved dramatically.[8]

Figure 1

The pursuit of continuous learning is aided by rapid changes in the quality and quantity of biomedical data available over the past decade, especially through the use of electronic health records, novel biomolecular tools, and digital sensors.[2, 7, 9] The Internet has ushered in a new era of data connectivity, for example, allowing for highly engaged communication between patients and providers as well as collaboration between professional or citizen scientists on data of unprecedented scale.[10] New methodologic approaches, including data mining and machine learning, increasingly leverage commodity hardware to conduct previously computationally intractable analyses.[9] Moreover, the development of domain ontologies fosters the discovery of meaningful insights from data of heterogeneous types.[11]

Ultimately, however, improvements in data alone are inadequate to achieve continuous learning. As shown in Figure 1, whereas data form the channels that allow for transitions from science to evidence to care, novel insights need to be steeped within the right culture, motivated by the right incentives, and supported by the right leaders.[1, 12] Within the sustainable learning healthcare system, knowledge generation feeds practice change with the support and guidance of system leadership; improved practice, in turn, generates new knowledge and completes the virtuous cycle of learning.

THE PROMISE OF CONTINUOUS LEARNING IN HOSPITAL SETTINGS

The hospital is an ideal setting in which to foster continuous learning because advances in inpatient care have the potential to substantially improve healthcare quality and value.[8] Americans were hospitalized roughly 37 million times in 2012; in total, these episodes cost $378 billion.[3] Over 700,000 patients die in US hospitals annually, with reports showing that many patients utilize greatly increased inpatient and critical care services near the end of their lives in a manner that appears misaligned with their preferences.[11, 13] Hospital care is also highly variable in quality and cost; this heterogeneity is not closely associated with improved outcomes.[14, 15] Preventable harm and medical injury occur commonly in hospitals and are now recognized to be a leading cause of inpatient death.[16] Finally, emerging research illuminates the substantial toll that acute care has on patients and families resulting in new comorbidity, functional or neuropsychiatric impairment, rehospitalization, and financial burden that persist long after patients are discharged.[17]

Fortunately, inpatient care also exhibits several qualities that improve the likelihood that continuous learning can be achieved. Although it is clear that hospitalizations occur within the arc of a patient's larger health trajectory, these distinct episodes offer the potential to observe patient trajectories and treatments evolving within relatively compressed time intervals; over that same interval, a large volume of data are produced. Stored within comprehensive electronic health records, these granular data now allow inpatient episodes to be digitally recapitulated with high fidelity, bolstering their use in driving care improvements.[18]

AN OBJECT LESSON IN THE LEARNING FRAMEWORK: SEPSIS CARE

Translating Science to Evidence in Sepsis

Although sepsis has attracted great attention in modern hospital care, sepsis was described long ago by Hippocrates to describe the process by which wounds fester.[19] Recast after the confirmation of germ theory, sepsis came to be known primarily as the blood poisoning resulting from pathogenic organisms.[20] However, with the advent of antibiotics, numerous scientific studies now recognize that sepsis actually results from the dysregulated host immune response to systemic infection, which can also cause organ dysfunction.[21] Based on this knowledge, landmark translational and clinical studies in the 2000s provided strong evidence that early identification of sepsis patients and aggressive infection control and resuscitation were associated with improved mortality (Figure 2, step 1).[22]

Figure 2

Translating Evidence to Care in Sepsis at KPNC

In 2007, the leadership of KPNC initiated a regional effort to improve the quality of care and reduce the variability in performance at its medical centers (Table 1).[23] Reviewing data from nearly 1000 inpatientsthe last 50 consecutive hospital deaths from each of 19 medical centersa mortality diagnostic based on Institute for Healthcare Improvement recommendations[24] revealed that sepsis had a major impact on hospital outcomes. For example, even though sepsis patients were still relatively under‐recognized at the time, accounting for fewer than 3% of hospitalizations, they contributed to one‐quarter of hospital deaths. In light of these compelling data, senior regional leadership identified reducing sepsis mortality as a key performance improvement goal (Figure 2, step 2).

Table 1.Timeline of Elements in the Kaiser Permanente Northern California Sepsis Performance Improvement Effort

Time Period	Event Summary
NOTE: Specific elements related to data infrastructure and analysis are indicated in italics. Abbreviations: ARISE, Australasian Resuscitation in Sepsis Evaluation; CMS SEP‐1, Centers for Medicare and Medicaide Services Severe Sepsis and Septic Shock: Management Bundle; ED, emergency department; EGDT, early goal‐directed therapy; KP, Kaiser Permanente; ProCESS, Protocolized Care for Early Septic Shock; ProMISe, Protocolized Care for Early Septic Shock.
2007	Variability in hospital standardized mortality ratio observed, indicating an opportunity to drive improved outcomes.
	Initiation of staggered implementation of unified electronic medical record across all KP sites (starting in 2006 and ending in 2009).
Spring 2008	Mortality diagnostic chart review completed identifying sepsis and infection‐related causes as key factors in hospital outcomes.
May 2008	Regional Mortality Summit held with a focus on patient safety and mortality reduction efforts through performance improvement. Executive regional and local leadership alignment to focus on sepsis performance improvement.
Summer 2008	Sepsis Steering Committee evaluates best available evidence, develops treatment algorithms, and plans for medical center pilots.
Fall 2008	Pilot intervention deployed at 2 medical centers.
November 2008	First Regional Sepsis Summit: development of sepsis performance improvement playbook, training materials, implementation plans, and measurement strategy.
November 2008	All medical centers begin to form multidisciplinary sepsis teams and performance improvement committees, obtain equipment and supplies including assembly of a sepsis cart. Multidisciplinary teams included ED physician champion, ED nurse champion, improvement advisor, hospitalists, intensivists, quality improvement personnel, nurse educators, and even resident physicians.
January 2009	Performance data collection begins on EGDT processes and outcomes. Initiation of 2 key elements to enhance screening for and detection of sepsis: (1) concomitant ordering of serum lactic acid along with blood cultures, and (2) definition of lactate >2.0 as a critical lab value.
	Use of manual chart review for case finding and central database entry because of ongoing implementation of electronic medical record and limited sepsis‐specific data infrastructure.
March 2009	Regional train the trainer sessions occur and local educational spread efforts begin including: collaborative calls, in‐person training events, and medical center site visits.
August 2009	Grant funding from the Gordon and Betty Moore Foundation begins with a planned 2‐year duration providing funding for improvement advisors with performance improvement expertise and data infrastructure development.
November 2009	Second Regional Sepsis Summit. Identification of intermediate lactate sepsis patients having significant mortality.
January 2010	Initiate measurement of performance for intermediate lactate sepsis patients with a focus on lactate clearance as an outcome measure of interest.
2010	Development of an intranet Web‐based data abstraction tool to identify cases and auto‐populate specific fields for review. Facilities were responsible for review of cases at the local level to foster rapid feedback cycles for local performance improvement. Standardized data query tools were deployed to foster local medical center engagement and system‐level evaluation.
	Accompanying development of a sepsis performance improvement scorecard allowing for comparison of longitudinal performance metrics across all facilities. Scorecard elements included: proportion of lactates drawn following ED blood culture, EGDT‐specific bundle elements (ie, number of EGDT cases, antibiotics within 1 hour, first central venous pressure within 2 hours of EGDT start, target mean arterial pressure achievement), repeat lactate elements, balancing measures for central line placement (ie, pneumothorax, central line infection), and overall sepsis statistics.
April 2011	Third Regional Sepsis Summit. Refinement of EGDT bundle and further development of intermediate lactate bundle approach, including piloting specific treatment bundles targeting this population. Collaborative performance improvement environment in which successful strategies at 1 site were rapidly disseminated to other sites including the Sepsis Alert and the Sepsis Clock.
May 2012	Research analysis of fluid volume and lactate clearance in intermediate lactate sepsis population begins.
February 2013	Fourth Regional Sepsis Summit. Regional spread of intermediate lactate bundle including the use of fluids, antibiotics, and repeat lactate measurements.
May 2013	Research analysis of the contribution of sepsis to hospital deaths (within KP and in national sample) as well as post‐sepsis resource utilization and mortality
March 2014	Publication of ProCESS randomized clinical trial, requiring systemic reevaluation of EGDT‐based sepsis strategy. Subsequent publications of ARISE and ProMISe trials confirming findings from ProCESS. Updated approach under consideration and informally disseminated to practitioners.
October 2014	Updated sepsis treatment guidelines and data capture strategy fully implemented moving away from a catheter‐based strategy for all EGDT‐eligible patients.
October 2015	Sixth Regional Sepsis Summit held to adjust sepsis treatment and data measurement strategy to align more closely with CMS SEP‐1 guidelines.

Based on the principles of performance improvement methodology, clinical and operational leaders established an environment with aligned culture, incentives, and leadership around sepsis care. The effort was launched in late 2008 at a Sepsis Summit, bringing together a multidisciplinary group of stakeholders (eg, hospitalist, emergency department, and intensive care chiefs of staff and nursing managers; medical center and nursing executive and operational leadership) and providing sepsis care pathways based on the best available evidence.[23] Regional investments in the digital infrastructure to support implementation resulted in the provision of granular data within monthly sepsis scorecards quantifying each medical center's performance and trends for a diverse set of sepsis bundle metrics.

The resulting changes in sepsis care were substantial. For example, improved early recognition of infected patients meeting the criteria for sepsis resulted in large changes in the standardized diagnostic criteria used to label patients (Figure 3A). Implementing screening strategies using serum lactate testing for any patient receiving blood cultures resulted in a roughly 10‐fold increase in the use of lactate testing in the emergency department (Figure 3B). Earlier recognition of sepsis also increased the number of patients receiving early antibiotics and receiving central venous catheters for quantitative resuscitation.[23]

Figure 3

CLOSING THE LOOP TOWARD CONTINUOUS LEARNING IN SEPSIS

Leveraging timely and actionable data steeped within an aligned organizational milieu resulted in large‐scale changes across a heterogeneous set of hospitals. However, to realize the true vision of a learning hospital system, a looming question remained: Could the data generated as the byproduct of routine care now be used to complete the virtuous cycle and drive new scientific discovery (Figure 2, step 3)?

PRINCIPLES FOR LEVERAGING DATA IN THE LEARNING HOSPITAL SYSTEM

The object lesson of using data to drive improved sepsis care and further new scientific discovery offers some important insights for continuous learning.

CONCLUSION

Hospitalizations in the United States are costly and associated with high mortality and morbidity; the toll of hospitalization also extends well beyond hospital discharge. The promise of the learning hospital system has marked improvements in the quality of hospital care, especially where healthcare systems can steep critical investments in data and digital infrastructure within the right culture, incentives, and leadership. Where continuous learning is achieved, data generated during routine care offer the potential to yield new scientific discovery and drive further improvements in hospital care.

In the landmark Best Care at Lower Cost report, the Institute of Medicine presents a compelling vision of a US healthcare system where science, information technology, incentives, and care culture are brought together seamlessly to produce high‐quality healthcare.[1] At the center of this transformation is the learning healthcare system, a system characterized by its ability to leverage data arising from care provision to drive rapid improvements in care delivery.[2] When steeped within the right organizational milieu, these data help to close the virtuous cycle of continuous learning moving from science to evidence to care and back to new science. The anticipated end result is a healthcare system that can provide Americans with superior care at lower cost.

Hospital‐based practitioners will recognize the inpatient setting as an ideal demonstration opportunity for continuous learning. Hospital care is costly, accounting for more than 30% of all US healthcare costs[3]; intensive care alone accounts for a notable proportion of the US gross domestic product.[4] Inpatient care is associated with significant mortality and morbidity, and its use is often greatly increased in patients' last days.[5, 6] Fortunately, the inpatient setting also offers an ideal opportunity to leverage high‐quality data to help inform and improve care. The digitization of medicine means that far more data are now available through electronic health records, medical devices, and tests.[7] This is particularly true for inpatients, for whom a large volume of data are produced even over relatively short hospital stays.

Whereas the challenge to improve hospital care is daunting, there is an incredible opportunity to advance the quality of inpatient care through realizing the vision of the learning hospital system. In the sections that follow, we use an object lessonsepsis care within hospitals of the Kaiser Permanente Northern California (KPNC) integrated healthcare delivery systemto evaluate the challenges and insights gleaned from working toward building a learning hospital system. Then, we describe further steps that could enhance the use of inpatient data to drive improved care.

THE FRAMEWORK OF A LEARNING HEALTHCARE SYSTEM

Best Care at Lower Cost notes a fundamental paradox in US healthcare: although we have witnessed a dramatic expansion in biomedical knowledge, innovative therapies and surgical procedures, and clinical treatments to extend survival, US healthcare persistently falls short on the basic dimensions of quality, outcomes, cost, and equity.[1] The proposed path forward lies in building the learning healthcare system, a system characterized by continuous knowledge development, improvement, and application. Figure 1 shows the critical nodes in the framework for continuous learning, which include: (1) the development of new scientific knowledge (science), (2) the translation of science into clinical evidence of efficacy (evidence), and (3) the application of efficacious interventions through effective care delivery (care). In healthcare today, transitions between these nodes are rife with missed or wasted opportunities like delays in applying high‐quality evidence or poorly managed insights arising from scientific discovery. If such opportunities could be recovered, however, the quality of healthcare could be improved dramatically.[8]

Figure 1

The pursuit of continuous learning is aided by rapid changes in the quality and quantity of biomedical data available over the past decade, especially through the use of electronic health records, novel biomolecular tools, and digital sensors.[2, 7, 9] The Internet has ushered in a new era of data connectivity, for example, allowing for highly engaged communication between patients and providers as well as collaboration between professional or citizen scientists on data of unprecedented scale.[10] New methodologic approaches, including data mining and machine learning, increasingly leverage commodity hardware to conduct previously computationally intractable analyses.[9] Moreover, the development of domain ontologies fosters the discovery of meaningful insights from data of heterogeneous types.[11]

Ultimately, however, improvements in data alone are inadequate to achieve continuous learning. As shown in Figure 1, whereas data form the channels that allow for transitions from science to evidence to care, novel insights need to be steeped within the right culture, motivated by the right incentives, and supported by the right leaders.[1, 12] Within the sustainable learning healthcare system, knowledge generation feeds practice change with the support and guidance of system leadership; improved practice, in turn, generates new knowledge and completes the virtuous cycle of learning.

THE PROMISE OF CONTINUOUS LEARNING IN HOSPITAL SETTINGS

The hospital is an ideal setting in which to foster continuous learning because advances in inpatient care have the potential to substantially improve healthcare quality and value.[8] Americans were hospitalized roughly 37 million times in 2012; in total, these episodes cost $378 billion.[3] Over 700,000 patients die in US hospitals annually, with reports showing that many patients utilize greatly increased inpatient and critical care services near the end of their lives in a manner that appears misaligned with their preferences.[11, 13] Hospital care is also highly variable in quality and cost; this heterogeneity is not closely associated with improved outcomes.[14, 15] Preventable harm and medical injury occur commonly in hospitals and are now recognized to be a leading cause of inpatient death.[16] Finally, emerging research illuminates the substantial toll that acute care has on patients and families resulting in new comorbidity, functional or neuropsychiatric impairment, rehospitalization, and financial burden that persist long after patients are discharged.[17]

Fortunately, inpatient care also exhibits several qualities that improve the likelihood that continuous learning can be achieved. Although it is clear that hospitalizations occur within the arc of a patient's larger health trajectory, these distinct episodes offer the potential to observe patient trajectories and treatments evolving within relatively compressed time intervals; over that same interval, a large volume of data are produced. Stored within comprehensive electronic health records, these granular data now allow inpatient episodes to be digitally recapitulated with high fidelity, bolstering their use in driving care improvements.[18]

AN OBJECT LESSON IN THE LEARNING FRAMEWORK: SEPSIS CARE

Translating Science to Evidence in Sepsis

Although sepsis has attracted great attention in modern hospital care, sepsis was described long ago by Hippocrates to describe the process by which wounds fester.[19] Recast after the confirmation of germ theory, sepsis came to be known primarily as the blood poisoning resulting from pathogenic organisms.[20] However, with the advent of antibiotics, numerous scientific studies now recognize that sepsis actually results from the dysregulated host immune response to systemic infection, which can also cause organ dysfunction.[21] Based on this knowledge, landmark translational and clinical studies in the 2000s provided strong evidence that early identification of sepsis patients and aggressive infection control and resuscitation were associated with improved mortality (Figure 2, step 1).[22]

Figure 2

Translating Evidence to Care in Sepsis at KPNC

In 2007, the leadership of KPNC initiated a regional effort to improve the quality of care and reduce the variability in performance at its medical centers (Table 1).[23] Reviewing data from nearly 1000 inpatientsthe last 50 consecutive hospital deaths from each of 19 medical centersa mortality diagnostic based on Institute for Healthcare Improvement recommendations[24] revealed that sepsis had a major impact on hospital outcomes. For example, even though sepsis patients were still relatively under‐recognized at the time, accounting for fewer than 3% of hospitalizations, they contributed to one‐quarter of hospital deaths. In light of these compelling data, senior regional leadership identified reducing sepsis mortality as a key performance improvement goal (Figure 2, step 2).

Table 1.Timeline of Elements in the Kaiser Permanente Northern California Sepsis Performance Improvement Effort

Time Period	Event Summary
NOTE: Specific elements related to data infrastructure and analysis are indicated in italics. Abbreviations: ARISE, Australasian Resuscitation in Sepsis Evaluation; CMS SEP‐1, Centers for Medicare and Medicaide Services Severe Sepsis and Septic Shock: Management Bundle; ED, emergency department; EGDT, early goal‐directed therapy; KP, Kaiser Permanente; ProCESS, Protocolized Care for Early Septic Shock; ProMISe, Protocolized Care for Early Septic Shock.
2007	Variability in hospital standardized mortality ratio observed, indicating an opportunity to drive improved outcomes.
	Initiation of staggered implementation of unified electronic medical record across all KP sites (starting in 2006 and ending in 2009).
Spring 2008	Mortality diagnostic chart review completed identifying sepsis and infection‐related causes as key factors in hospital outcomes.
May 2008	Regional Mortality Summit held with a focus on patient safety and mortality reduction efforts through performance improvement. Executive regional and local leadership alignment to focus on sepsis performance improvement.
Summer 2008	Sepsis Steering Committee evaluates best available evidence, develops treatment algorithms, and plans for medical center pilots.
Fall 2008	Pilot intervention deployed at 2 medical centers.
November 2008	First Regional Sepsis Summit: development of sepsis performance improvement playbook, training materials, implementation plans, and measurement strategy.
November 2008	All medical centers begin to form multidisciplinary sepsis teams and performance improvement committees, obtain equipment and supplies including assembly of a sepsis cart. Multidisciplinary teams included ED physician champion, ED nurse champion, improvement advisor, hospitalists, intensivists, quality improvement personnel, nurse educators, and even resident physicians.
January 2009	Performance data collection begins on EGDT processes and outcomes. Initiation of 2 key elements to enhance screening for and detection of sepsis: (1) concomitant ordering of serum lactic acid along with blood cultures, and (2) definition of lactate >2.0 as a critical lab value.
	Use of manual chart review for case finding and central database entry because of ongoing implementation of electronic medical record and limited sepsis‐specific data infrastructure.
March 2009	Regional train the trainer sessions occur and local educational spread efforts begin including: collaborative calls, in‐person training events, and medical center site visits.
August 2009	Grant funding from the Gordon and Betty Moore Foundation begins with a planned 2‐year duration providing funding for improvement advisors with performance improvement expertise and data infrastructure development.
November 2009	Second Regional Sepsis Summit. Identification of intermediate lactate sepsis patients having significant mortality.
January 2010	Initiate measurement of performance for intermediate lactate sepsis patients with a focus on lactate clearance as an outcome measure of interest.
2010	Development of an intranet Web‐based data abstraction tool to identify cases and auto‐populate specific fields for review. Facilities were responsible for review of cases at the local level to foster rapid feedback cycles for local performance improvement. Standardized data query tools were deployed to foster local medical center engagement and system‐level evaluation.
	Accompanying development of a sepsis performance improvement scorecard allowing for comparison of longitudinal performance metrics across all facilities. Scorecard elements included: proportion of lactates drawn following ED blood culture, EGDT‐specific bundle elements (ie, number of EGDT cases, antibiotics within 1 hour, first central venous pressure within 2 hours of EGDT start, target mean arterial pressure achievement), repeat lactate elements, balancing measures for central line placement (ie, pneumothorax, central line infection), and overall sepsis statistics.
April 2011	Third Regional Sepsis Summit. Refinement of EGDT bundle and further development of intermediate lactate bundle approach, including piloting specific treatment bundles targeting this population. Collaborative performance improvement environment in which successful strategies at 1 site were rapidly disseminated to other sites including the Sepsis Alert and the Sepsis Clock.
May 2012	Research analysis of fluid volume and lactate clearance in intermediate lactate sepsis population begins.
February 2013	Fourth Regional Sepsis Summit. Regional spread of intermediate lactate bundle including the use of fluids, antibiotics, and repeat lactate measurements.
May 2013	Research analysis of the contribution of sepsis to hospital deaths (within KP and in national sample) as well as post‐sepsis resource utilization and mortality
March 2014	Publication of ProCESS randomized clinical trial, requiring systemic reevaluation of EGDT‐based sepsis strategy. Subsequent publications of ARISE and ProMISe trials confirming findings from ProCESS. Updated approach under consideration and informally disseminated to practitioners.
October 2014	Updated sepsis treatment guidelines and data capture strategy fully implemented moving away from a catheter‐based strategy for all EGDT‐eligible patients.
October 2015	Sixth Regional Sepsis Summit held to adjust sepsis treatment and data measurement strategy to align more closely with CMS SEP‐1 guidelines.

Based on the principles of performance improvement methodology, clinical and operational leaders established an environment with aligned culture, incentives, and leadership around sepsis care. The effort was launched in late 2008 at a Sepsis Summit, bringing together a multidisciplinary group of stakeholders (eg, hospitalist, emergency department, and intensive care chiefs of staff and nursing managers; medical center and nursing executive and operational leadership) and providing sepsis care pathways based on the best available evidence.[23] Regional investments in the digital infrastructure to support implementation resulted in the provision of granular data within monthly sepsis scorecards quantifying each medical center's performance and trends for a diverse set of sepsis bundle metrics.

The resulting changes in sepsis care were substantial. For example, improved early recognition of infected patients meeting the criteria for sepsis resulted in large changes in the standardized diagnostic criteria used to label patients (Figure 3A). Implementing screening strategies using serum lactate testing for any patient receiving blood cultures resulted in a roughly 10‐fold increase in the use of lactate testing in the emergency department (Figure 3B). Earlier recognition of sepsis also increased the number of patients receiving early antibiotics and receiving central venous catheters for quantitative resuscitation.[23]

Figure 3

CLOSING THE LOOP TOWARD CONTINUOUS LEARNING IN SEPSIS

Leveraging timely and actionable data steeped within an aligned organizational milieu resulted in large‐scale changes across a heterogeneous set of hospitals. However, to realize the true vision of a learning hospital system, a looming question remained: Could the data generated as the byproduct of routine care now be used to complete the virtuous cycle and drive new scientific discovery (Figure 2, step 3)?

PRINCIPLES FOR LEVERAGING DATA IN THE LEARNING HOSPITAL SYSTEM

The object lesson of using data to drive improved sepsis care and further new scientific discovery offers some important insights for continuous learning.

CONCLUSION

Hospitalizations in the United States are costly and associated with high mortality and morbidity; the toll of hospitalization also extends well beyond hospital discharge. The promise of the learning hospital system has marked improvements in the quality of hospital care, especially where healthcare systems can steep critical investments in data and digital infrastructure within the right culture, incentives, and leadership. Where continuous learning is achieved, data generated during routine care offer the potential to yield new scientific discovery and drive further improvements in hospital care.

In the landmark Best Care at Lower Cost report, the Institute of Medicine presents a compelling vision of a US healthcare system where science, information technology, incentives, and care culture are brought together seamlessly to produce high‐quality healthcare.[1] At the center of this transformation is the learning healthcare system, a system characterized by its ability to leverage data arising from care provision to drive rapid improvements in care delivery.[2] When steeped within the right organizational milieu, these data help to close the virtuous cycle of continuous learning moving from science to evidence to care and back to new science. The anticipated end result is a healthcare system that can provide Americans with superior care at lower cost.

Hospital‐based practitioners will recognize the inpatient setting as an ideal demonstration opportunity for continuous learning. Hospital care is costly, accounting for more than 30% of all US healthcare costs[3]; intensive care alone accounts for a notable proportion of the US gross domestic product.[4] Inpatient care is associated with significant mortality and morbidity, and its use is often greatly increased in patients' last days.[5, 6] Fortunately, the inpatient setting also offers an ideal opportunity to leverage high‐quality data to help inform and improve care. The digitization of medicine means that far more data are now available through electronic health records, medical devices, and tests.[7] This is particularly true for inpatients, for whom a large volume of data are produced even over relatively short hospital stays.

Whereas the challenge to improve hospital care is daunting, there is an incredible opportunity to advance the quality of inpatient care through realizing the vision of the learning hospital system. In the sections that follow, we use an object lessonsepsis care within hospitals of the Kaiser Permanente Northern California (KPNC) integrated healthcare delivery systemto evaluate the challenges and insights gleaned from working toward building a learning hospital system. Then, we describe further steps that could enhance the use of inpatient data to drive improved care.

THE FRAMEWORK OF A LEARNING HEALTHCARE SYSTEM

Best Care at Lower Cost notes a fundamental paradox in US healthcare: although we have witnessed a dramatic expansion in biomedical knowledge, innovative therapies and surgical procedures, and clinical treatments to extend survival, US healthcare persistently falls short on the basic dimensions of quality, outcomes, cost, and equity.[1] The proposed path forward lies in building the learning healthcare system, a system characterized by continuous knowledge development, improvement, and application. Figure 1 shows the critical nodes in the framework for continuous learning, which include: (1) the development of new scientific knowledge (science), (2) the translation of science into clinical evidence of efficacy (evidence), and (3) the application of efficacious interventions through effective care delivery (care). In healthcare today, transitions between these nodes are rife with missed or wasted opportunities like delays in applying high‐quality evidence or poorly managed insights arising from scientific discovery. If such opportunities could be recovered, however, the quality of healthcare could be improved dramatically.[8]

Figure 1

The pursuit of continuous learning is aided by rapid changes in the quality and quantity of biomedical data available over the past decade, especially through the use of electronic health records, novel biomolecular tools, and digital sensors.[2, 7, 9] The Internet has ushered in a new era of data connectivity, for example, allowing for highly engaged communication between patients and providers as well as collaboration between professional or citizen scientists on data of unprecedented scale.[10] New methodologic approaches, including data mining and machine learning, increasingly leverage commodity hardware to conduct previously computationally intractable analyses.[9] Moreover, the development of domain ontologies fosters the discovery of meaningful insights from data of heterogeneous types.[11]

Ultimately, however, improvements in data alone are inadequate to achieve continuous learning. As shown in Figure 1, whereas data form the channels that allow for transitions from science to evidence to care, novel insights need to be steeped within the right culture, motivated by the right incentives, and supported by the right leaders.[1, 12] Within the sustainable learning healthcare system, knowledge generation feeds practice change with the support and guidance of system leadership; improved practice, in turn, generates new knowledge and completes the virtuous cycle of learning.

THE PROMISE OF CONTINUOUS LEARNING IN HOSPITAL SETTINGS

The hospital is an ideal setting in which to foster continuous learning because advances in inpatient care have the potential to substantially improve healthcare quality and value.[8] Americans were hospitalized roughly 37 million times in 2012; in total, these episodes cost $378 billion.[3] Over 700,000 patients die in US hospitals annually, with reports showing that many patients utilize greatly increased inpatient and critical care services near the end of their lives in a manner that appears misaligned with their preferences.[11, 13] Hospital care is also highly variable in quality and cost; this heterogeneity is not closely associated with improved outcomes.[14, 15] Preventable harm and medical injury occur commonly in hospitals and are now recognized to be a leading cause of inpatient death.[16] Finally, emerging research illuminates the substantial toll that acute care has on patients and families resulting in new comorbidity, functional or neuropsychiatric impairment, rehospitalization, and financial burden that persist long after patients are discharged.[17]

Fortunately, inpatient care also exhibits several qualities that improve the likelihood that continuous learning can be achieved. Although it is clear that hospitalizations occur within the arc of a patient's larger health trajectory, these distinct episodes offer the potential to observe patient trajectories and treatments evolving within relatively compressed time intervals; over that same interval, a large volume of data are produced. Stored within comprehensive electronic health records, these granular data now allow inpatient episodes to be digitally recapitulated with high fidelity, bolstering their use in driving care improvements.[18]

AN OBJECT LESSON IN THE LEARNING FRAMEWORK: SEPSIS CARE

Translating Science to Evidence in Sepsis

Although sepsis has attracted great attention in modern hospital care, sepsis was described long ago by Hippocrates to describe the process by which wounds fester.[19] Recast after the confirmation of germ theory, sepsis came to be known primarily as the blood poisoning resulting from pathogenic organisms.[20] However, with the advent of antibiotics, numerous scientific studies now recognize that sepsis actually results from the dysregulated host immune response to systemic infection, which can also cause organ dysfunction.[21] Based on this knowledge, landmark translational and clinical studies in the 2000s provided strong evidence that early identification of sepsis patients and aggressive infection control and resuscitation were associated with improved mortality (Figure 2, step 1).[22]

Figure 2

Translating Evidence to Care in Sepsis at KPNC

In 2007, the leadership of KPNC initiated a regional effort to improve the quality of care and reduce the variability in performance at its medical centers (Table 1).[23] Reviewing data from nearly 1000 inpatientsthe last 50 consecutive hospital deaths from each of 19 medical centersa mortality diagnostic based on Institute for Healthcare Improvement recommendations[24] revealed that sepsis had a major impact on hospital outcomes. For example, even though sepsis patients were still relatively under‐recognized at the time, accounting for fewer than 3% of hospitalizations, they contributed to one‐quarter of hospital deaths. In light of these compelling data, senior regional leadership identified reducing sepsis mortality as a key performance improvement goal (Figure 2, step 2).

Table 1.Timeline of Elements in the Kaiser Permanente Northern California Sepsis Performance Improvement Effort

Time Period	Event Summary
NOTE: Specific elements related to data infrastructure and analysis are indicated in italics. Abbreviations: ARISE, Australasian Resuscitation in Sepsis Evaluation; CMS SEP‐1, Centers for Medicare and Medicaide Services Severe Sepsis and Septic Shock: Management Bundle; ED, emergency department; EGDT, early goal‐directed therapy; KP, Kaiser Permanente; ProCESS, Protocolized Care for Early Septic Shock; ProMISe, Protocolized Care for Early Septic Shock.
2007	Variability in hospital standardized mortality ratio observed, indicating an opportunity to drive improved outcomes.
	Initiation of staggered implementation of unified electronic medical record across all KP sites (starting in 2006 and ending in 2009).
Spring 2008	Mortality diagnostic chart review completed identifying sepsis and infection‐related causes as key factors in hospital outcomes.
May 2008	Regional Mortality Summit held with a focus on patient safety and mortality reduction efforts through performance improvement. Executive regional and local leadership alignment to focus on sepsis performance improvement.
Summer 2008	Sepsis Steering Committee evaluates best available evidence, develops treatment algorithms, and plans for medical center pilots.
Fall 2008	Pilot intervention deployed at 2 medical centers.
November 2008	First Regional Sepsis Summit: development of sepsis performance improvement playbook, training materials, implementation plans, and measurement strategy.
November 2008	All medical centers begin to form multidisciplinary sepsis teams and performance improvement committees, obtain equipment and supplies including assembly of a sepsis cart. Multidisciplinary teams included ED physician champion, ED nurse champion, improvement advisor, hospitalists, intensivists, quality improvement personnel, nurse educators, and even resident physicians.
January 2009	Performance data collection begins on EGDT processes and outcomes. Initiation of 2 key elements to enhance screening for and detection of sepsis: (1) concomitant ordering of serum lactic acid along with blood cultures, and (2) definition of lactate >2.0 as a critical lab value.
	Use of manual chart review for case finding and central database entry because of ongoing implementation of electronic medical record and limited sepsis‐specific data infrastructure.
March 2009	Regional train the trainer sessions occur and local educational spread efforts begin including: collaborative calls, in‐person training events, and medical center site visits.
August 2009	Grant funding from the Gordon and Betty Moore Foundation begins with a planned 2‐year duration providing funding for improvement advisors with performance improvement expertise and data infrastructure development.
November 2009	Second Regional Sepsis Summit. Identification of intermediate lactate sepsis patients having significant mortality.
January 2010	Initiate measurement of performance for intermediate lactate sepsis patients with a focus on lactate clearance as an outcome measure of interest.
2010	Development of an intranet Web‐based data abstraction tool to identify cases and auto‐populate specific fields for review. Facilities were responsible for review of cases at the local level to foster rapid feedback cycles for local performance improvement. Standardized data query tools were deployed to foster local medical center engagement and system‐level evaluation.
	Accompanying development of a sepsis performance improvement scorecard allowing for comparison of longitudinal performance metrics across all facilities. Scorecard elements included: proportion of lactates drawn following ED blood culture, EGDT‐specific bundle elements (ie, number of EGDT cases, antibiotics within 1 hour, first central venous pressure within 2 hours of EGDT start, target mean arterial pressure achievement), repeat lactate elements, balancing measures for central line placement (ie, pneumothorax, central line infection), and overall sepsis statistics.
April 2011	Third Regional Sepsis Summit. Refinement of EGDT bundle and further development of intermediate lactate bundle approach, including piloting specific treatment bundles targeting this population. Collaborative performance improvement environment in which successful strategies at 1 site were rapidly disseminated to other sites including the Sepsis Alert and the Sepsis Clock.
May 2012	Research analysis of fluid volume and lactate clearance in intermediate lactate sepsis population begins.
February 2013	Fourth Regional Sepsis Summit. Regional spread of intermediate lactate bundle including the use of fluids, antibiotics, and repeat lactate measurements.
May 2013	Research analysis of the contribution of sepsis to hospital deaths (within KP and in national sample) as well as post‐sepsis resource utilization and mortality
March 2014	Publication of ProCESS randomized clinical trial, requiring systemic reevaluation of EGDT‐based sepsis strategy. Subsequent publications of ARISE and ProMISe trials confirming findings from ProCESS. Updated approach under consideration and informally disseminated to practitioners.
October 2014	Updated sepsis treatment guidelines and data capture strategy fully implemented moving away from a catheter‐based strategy for all EGDT‐eligible patients.
October 2015	Sixth Regional Sepsis Summit held to adjust sepsis treatment and data measurement strategy to align more closely with CMS SEP‐1 guidelines.

Based on the principles of performance improvement methodology, clinical and operational leaders established an environment with aligned culture, incentives, and leadership around sepsis care. The effort was launched in late 2008 at a Sepsis Summit, bringing together a multidisciplinary group of stakeholders (eg, hospitalist, emergency department, and intensive care chiefs of staff and nursing managers; medical center and nursing executive and operational leadership) and providing sepsis care pathways based on the best available evidence.[23] Regional investments in the digital infrastructure to support implementation resulted in the provision of granular data within monthly sepsis scorecards quantifying each medical center's performance and trends for a diverse set of sepsis bundle metrics.

The resulting changes in sepsis care were substantial. For example, improved early recognition of infected patients meeting the criteria for sepsis resulted in large changes in the standardized diagnostic criteria used to label patients (Figure 3A). Implementing screening strategies using serum lactate testing for any patient receiving blood cultures resulted in a roughly 10‐fold increase in the use of lactate testing in the emergency department (Figure 3B). Earlier recognition of sepsis also increased the number of patients receiving early antibiotics and receiving central venous catheters for quantitative resuscitation.[23]

Figure 3

CLOSING THE LOOP TOWARD CONTINUOUS LEARNING IN SEPSIS

Leveraging timely and actionable data steeped within an aligned organizational milieu resulted in large‐scale changes across a heterogeneous set of hospitals. However, to realize the true vision of a learning hospital system, a looming question remained: Could the data generated as the byproduct of routine care now be used to complete the virtuous cycle and drive new scientific discovery (Figure 2, step 3)?

PRINCIPLES FOR LEVERAGING DATA IN THE LEARNING HOSPITAL SYSTEM

The object lesson of using data to drive improved sepsis care and further new scientific discovery offers some important insights for continuous learning.

CONCLUSION

Hospitalizations in the United States are costly and associated with high mortality and morbidity; the toll of hospitalization also extends well beyond hospital discharge. The promise of the learning hospital system has marked improvements in the quality of hospital care, especially where healthcare systems can steep critical investments in data and digital infrastructure within the right culture, incentives, and leadership. Where continuous learning is achieved, data generated during routine care offer the potential to yield new scientific discovery and drive further improvements in hospital care.

If you wish to receive credit for this activity, which begins on the next page, please refer to the website: www.blackwellpublishing.com/cme.

Accreditation and Designation Statement

Blackwell Futura Media Services designates this educational activity for a 1 AMA PRA Category 1 Credit. Physicians should only claim credit commensurate with the extent of their participation in the activity.

Blackwell Futura Media Services is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians.

Educational Objectives

Upon completion of this educational activity, participants will be better able to employ automated bed history data to examine outcomes of intra‐hospital transfers using all hospital admissions as the denominator.

Continuous participation in the Journal of Hospital Medicine CME program will enable learners to be better able to:

• Interpret clinical guidelines and their applications for higher quality and more efficient care for all hospitalized patients.

• Describe the standard of care for common illnesses and conditions treated in the hospital; such as pneumonia, COPD exacerbation, acute coronary syndrome, HF exacerbation, glycemic control, venous thromboembolic disease, stroke, etc.

• Discuss evidence‐based recommendations involving transitions of care, including the hospital discharge process.

• Gain insights into the roles of hospitalists as medical educators, researchers, medical ethicists, palliative care providers, and hospital‐based geriatricians.

• Incorporate best practices for hospitalist administration, including quality improvement, patient safety, practice management, leadership, and demonstrating hospitalist value.

• Identify evidence‐based best practices and trends for both adult and pediatric hospital medicine.

Instructions on Receiving Credit

For information on applicability and acceptance of continuing medical education credit for this activity, please consult your professional licensing board.

This activity is designed to be completed within the time designated on the title page; physicians should claim only those credits that reflect the time actually spent in the activity. To successfully earn credit, participants must complete the activity during the valid credit period that is noted on the title page.

Follow these steps to earn credit:

• Log on to www.blackwellpublishing.com/cme.

• Read the target audience, learning objectives, and author disclosures.

• Read the article in print or online format.

• Reflect on the article.

• Access the CME Exam, and choose the best answer to each question.

• Complete the required evaluation component of the activity.

If you wish to receive credit for this activity, which begins on the next page, please refer to the website: www.blackwellpublishing.com/cme.

Accreditation and Designation Statement

Blackwell Futura Media Services designates this educational activity for a 1 AMA PRA Category 1 Credit. Physicians should only claim credit commensurate with the extent of their participation in the activity.

Blackwell Futura Media Services is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians.

Educational Objectives

Upon completion of this educational activity, participants will be better able to employ automated bed history data to examine outcomes of intra‐hospital transfers using all hospital admissions as the denominator.

Continuous participation in the Journal of Hospital Medicine CME program will enable learners to be better able to:

• Interpret clinical guidelines and their applications for higher quality and more efficient care for all hospitalized patients.

• Describe the standard of care for common illnesses and conditions treated in the hospital; such as pneumonia, COPD exacerbation, acute coronary syndrome, HF exacerbation, glycemic control, venous thromboembolic disease, stroke, etc.

• Discuss evidence‐based recommendations involving transitions of care, including the hospital discharge process.

• Gain insights into the roles of hospitalists as medical educators, researchers, medical ethicists, palliative care providers, and hospital‐based geriatricians.

• Incorporate best practices for hospitalist administration, including quality improvement, patient safety, practice management, leadership, and demonstrating hospitalist value.

• Identify evidence‐based best practices and trends for both adult and pediatric hospital medicine.

Instructions on Receiving Credit

For information on applicability and acceptance of continuing medical education credit for this activity, please consult your professional licensing board.

This activity is designed to be completed within the time designated on the title page; physicians should claim only those credits that reflect the time actually spent in the activity. To successfully earn credit, participants must complete the activity during the valid credit period that is noted on the title page.

Follow these steps to earn credit:

• Log on to www.blackwellpublishing.com/cme.

• Read the target audience, learning objectives, and author disclosures.

• Read the article in print or online format.

• Reflect on the article.

• Access the CME Exam, and choose the best answer to each question.

• Complete the required evaluation component of the activity.

If you wish to receive credit for this activity, which begins on the next page, please refer to the website: www.blackwellpublishing.com/cme.

Accreditation and Designation Statement

Blackwell Futura Media Services designates this educational activity for a 1 AMA PRA Category 1 Credit. Physicians should only claim credit commensurate with the extent of their participation in the activity.

Blackwell Futura Media Services is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians.

Educational Objectives

Upon completion of this educational activity, participants will be better able to employ automated bed history data to examine outcomes of intra‐hospital transfers using all hospital admissions as the denominator.

Continuous participation in the Journal of Hospital Medicine CME program will enable learners to be better able to:

• Interpret clinical guidelines and their applications for higher quality and more efficient care for all hospitalized patients.

• Describe the standard of care for common illnesses and conditions treated in the hospital; such as pneumonia, COPD exacerbation, acute coronary syndrome, HF exacerbation, glycemic control, venous thromboembolic disease, stroke, etc.

• Discuss evidence‐based recommendations involving transitions of care, including the hospital discharge process.

• Gain insights into the roles of hospitalists as medical educators, researchers, medical ethicists, palliative care providers, and hospital‐based geriatricians.

• Incorporate best practices for hospitalist administration, including quality improvement, patient safety, practice management, leadership, and demonstrating hospitalist value.

• Identify evidence‐based best practices and trends for both adult and pediatric hospital medicine.

Instructions on Receiving Credit

For information on applicability and acceptance of continuing medical education credit for this activity, please consult your professional licensing board.

This activity is designed to be completed within the time designated on the title page; physicians should claim only those credits that reflect the time actually spent in the activity. To successfully earn credit, participants must complete the activity during the valid credit period that is noted on the title page.

Follow these steps to earn credit:

• Log on to www.blackwellpublishing.com/cme.

• Read the target audience, learning objectives, and author disclosures.

• Read the article in print or online format.

• Reflect on the article.

• Access the CME Exam, and choose the best answer to each question.

• Complete the required evaluation component of the activity.

Considerable research and public attention is being paid to the quantification, risk adjustment, and reporting of inpatient mortality.15 Inpatient mortality is reported as aggregate mortality (for all hospitalized patients or those with a specific diagnosis3, 6) or intensive care unit (ICU) mortality.7, 8 While reporting aggregate hospital or aggregate ICU mortality rates is useful, it is also important to develop reporting strategies that go beyond simply using data elements found in administrative databases (eg, diagnosis and procedure codes) to quantify practice variation. Ideally, such strategies would permit delineating processes of careparticularly those potentially under the control of hospitalists, not only intensiviststo identify improvement opportunities. One such process, which can be tracked using the bed history component of a patient's electronic medical record, is the transfer of patients between different units within the same hospital.

Several studies have documented that risk of ICU death is highest among patients transferred from general medical‐surgical wards, intermediate among direct admissions from the emergency department, and lowest among surgical admissions.911 Opportunities to reduce subsequent ICU mortality have been studied among ward patients who develop sepsis and are then transferred to the ICU,12 among patients who experience cardiac arrest,13, 14 as well as among patients with any physiological deterioration (eg, through the use of rapid response teams).1517 Most of these studies have been single‐center studies and/or studies reporting only an ICU denominator. While useful in some respects, such studies are less helpful to hospitalists, who would benefit from better understanding of the types of patients transferred and the total impact that transfers to a higher level of care make on general medical‐surgical wards. In addition, entities such as the Institute for Healthcare Improvement recommend the manual review of records of patients who were transferred from the ward to the ICU18 to identify performance improvement opportunities. While laudable, such approaches do not lend themselves to automated reporting strategies.

We recently described a new risk adjustment methodology for inpatient mortality based entirely on automated data preceding hospital admission and not restricted to ICU patients. This methodology, which has been externally validated in Ottawa, Canada, after development in the Kaiser Permanente Medical Care Program (KPMCP), permits quantification of a patient's pre‐existing comorbidity burden, physiologic derangement at the time of admission, and overall inpatient mortality risk.19, 20 The primary purpose of this study was to combine this methodology with bed history analysis to quantify the in‐hospital mortality and length of stay (LOS) of patients who experienced intra‐hospital transfers in a large, multihospital system. As a secondary goal, we also wanted to assess the degree to which these transfers could be predicted based on information available prior to a patient's admission.

Materials and Methods

This project was approved by the Northern California KPMCP Institutional Review Board for the Protection of Human Subjects.

The Northern California KPMCP serves a total population of approximately 3.3 million members. Under a mutual exclusivity arrangement, physicians of The Permanente Medical Group, Inc., care for Kaiser Foundation Health Plan, Inc. members at facilities owned by Kaiser Foundation Hospitals, Inc. All Northern California KPMCP hospitals and clinics employ the same information systems with a common medical record number and can track care covered by the plan but delivered elsewhere. Databases maintained by the KPMCP capture admission and discharge times, admission and discharge diagnoses and procedures (assigned by professional coders), bed histories, inter‐hospital transfers, as well as the results of all inpatient and outpatient laboratory tests. The use of these databases for research has been described in multiple reports.2124

Our setting consisted of all 19 hospitals owned and operated by the KPMCP, whose characteristics are summarized in the Supporting Information Appendix available to interested readers. These include the 17 described in our previous report19 as well as 2 new hospitals (Antioch and Manteca) which are similar in size and type of population served. Our study population consisted of all patients admitted to these 19 hospitals who met these criteria: 1) hospitalization began from November 1st, 2006 through January 31st, 2008; 2) initial hospitalization occurred at a Northern California KPMCP hospital (ie, for inter‐hospital transfers, the first hospital stay occurred within the KPMCP); 3) age 15 years; and 4) hospitalization was not for childbirth.

We defined a linked hospitalization as the time period that began with a patient's admission to the hospital and ended with the patient's discharge (home, to a nursing home, or death). Linked hospitalizations can thus involve more than 1 hospital stay and could include a patient transfer from one hospital to another prior to definitive discharge. For linked hospitalizations, mortality was attributed to the admitting KPMCP hospital (ie, if a patient was admitted to hospital A, transferred to B, and died at hospital B, mortality was attributed to hospital A). We defined total LOS as the exact time in hours from when a patient was first admitted to the hospital until death or final discharge home or to a nursing home, while total ICU or transitional care unit (TCU, referred to as stepdown unit in some hospitals) LOS was calculated for all individual ICU or TCU stays during the hospital stay.

Results

During the study period, a total of 249,129 individual hospital stays involving 170,151 patients occurred at these 19 hospitals. After concatenation of inter‐hospital transfers, we were left with 237,208 linked hospitalizations. We excluded 26,738 linked hospitalizations that began at a non‐KPMCP hospital (ie, they were transported in), leaving a total of 210,470 linked hospitalizations involving 150,495 patients. The overall linked hospitalization mortality rate was 3.30%.

Table 1 summarizes cohort characteristics based on initial hospital location. On admission, ICU patients had the highest degree of physiologic derangement as well as the highest predicted mortality. Considerable inter‐hospital variation was present in both predictors and outcomes; details on these variations are provided in the Supporting Information Appendix.

Table 2 summarizes data from 3 groups of patients: patients initially admitted to the ward, or TCU, who did not experience a transfer to a higher level of care and patients admitted to these 2 units who did experience such a transfer. Patients who experienced a transfer constituted 5.3% (6,484/121,237) of ward patients and 6.7% (1,384/20,556) of TCU patients. Transferred patients tended to be older, have more acute physiologic derangement (higher LAPS), a greater pre‐existing illness burden (higher COPS), and a higher predicted mortality risk. Among ward patients, those with the following admission diagnoses were most likely to experience a transfer to a higher level of care: gastrointestinal bleeding (10.8% of all transfers), pneumonia (8.7%), and other infections (8.2%). The diagnoses most likely to be associated with death following transfer were cancer (death rate among transferred patients, 48%), renal disease (death rate, 36%), and liver disease (33%). Similar distributions were observed for TCU patients.

Table 3 compares outcomes among ward and TCU patients who did and did not experience a transfer to a higher level of care. The table shows that transferred patients were almost 3 times as likely to die, even after controlling for severity of illness, and that their hospital LOS was 9 days higher than expected. This increased risk was seen in all hospitals and among all transfer types (ward to ICU, ward to TCU, and TCU to ICU).

Table 3 also shows that, among decedent patients, those who never left the ward or TCU died much sooner than those who died following transfer. Among direct admits to the ICU, the median LOS at time of death was 3.9 days, with a mean of 9.4 standard deviation of 19.9 days, while the corresponding times for TCU direct admits were a median and mean LOS of 6.5 and 11.7 19.5 days.

Table 4 summarizes outcomes among different patient subgroups that did and did not experience a transfer to a higher level of care. Based on location, patients who experienced a transfer from the TCU to the ICU had the highest crude death rate, but patients transferred from the ward to the ICU had the highest OEMR. On the other hand, if one divides patients by the degree of physiologic derangement, patients with low LAPS who experienced a transfer had the highest OEMR. With respect to LOS, patients transferred from the TCU to the ICU had the highest OMELOS (13.4 extra days).

Transfers to a higher level of care at a different hospital, which in the KPMCP are usually planned, experienced lower mortality than transfers within the same hospital. For ward to TCU transfers, intra‐hospital transfers had a mortality of 12.1% while inter‐hospital transfers had a mortality of 5.7%. Corresponding rates for ward to ICU transfers were 21.7% and 11.2%, and for TCU to ICU transfers the rates were 25.9% and 12.5%, respectively.

Among patients initially admitted to the ward, a model to predict the occurrence of a transfer to a higher level of care (within 48 hours after admission) that included age, sex, admission type, primary condition, LAPS, COPS, and interaction terms had poor discrimination, with an area under the receiver operator characteristic (c statistic) of only 0.64. The c statistic for a model to predict transfer after 48 hours was 0.66. The corresponding models for TCU admits had c statistics of 0.67 and 0.68. All four models had poor calibration.

Discussion

Using automated bed history data permits characterizing a patient population with disproportionate mortality and LOS: intra‐hospital transfers to special care units (ICUs or TCUs). Indeed, the largest subset of these patients (those initially admitted to the ward or TCU) constituted only 3.7% of all admissions, but accounted for 24.2% of all ICU admissions, 21.7% of all hospital deaths, and 13.2% of all hospital days. These patients also had very elevated OEMRs and OMELOS. Models based on age, sex, preadmission laboratory test results, and comorbidities did not predict the occurrence of these transfers.

We performed multivariate analyses to explore the degree to which electronically assigned preadmission severity scores could predict these transfers. These analyses found that, compared to our ability to predict inpatient or 30‐day mortality at the time of admission, which is excellent, our ability to predict the occurrence of transfer after admission is much more limited. These results highlight the limitations of severity scores that rely on automated data, which may not have adequate discrimination when it comes to determining the risk of an adverse outcome within a narrow time frame. For example, among the 121,237 patients initially admitted to the ward who did not experience an intra‐hospital transfer, the mean LAPS was 18.9, while the mean LAPS among the 6,484 ward patients who did experience a transfer was 25.5. Differences between the mean and median LAPS, COPS, and predicted mortality risk among transferred and non‐transferred patients were significant (P < 0.0001 for all comparisons). However, examination of the distribution of LAPS, COPS, and predicted mortality risk between these two groups of patients showed considerable overlap.

Our methodology resembles Silber et al.'s27, 28 concept of failure to rescue in that it focuses on events occurring after hospitalization. Silber et al. argue that a hospital's quality can be measured by quantifying the degree to which patients who experience new problems are successfully rescued. Furthermore, quantification of those situations where rescue attempts are unsuccessful is felt to be superior to simply comparing raw or adjusted mortality rates because these are primarily determined by underlying case mix. The primary difference between Silber et al.'s approach and ours is at the level of detailthey specified a specific set of complications, whereas our measure is more generic and would include patients with many of the complications specified by Silber et al.27, 28

Most of the patients transferred to a higher level of care in our cohort survived (ie, were rescued), indicating that intensive care is beneficial. However, the fact that these patients had elevated OEMRs and OMELOS indicates that the real challenge facing hospitalists involves the timing of provision of a beneficial intervention. In theory, improved timing could result from earlier detection of problems, which is the underlying rationale for employing rapid response teams. However, the fact that our electronic tools (LAPS, COPS) cannot predict patient deteriorations within a narrow time frame suggests that early detection will remain a major challenge. Manually assigned vital signs scores designed for this purpose do not have good discrimination either.29, 30 This raises the possibility that, though patient groups may differ in terms of overall illness severity and mortality risk, differences at the individual patient level may be too subtle for clinicians to detect. Future research may thus need to focus on scores that combine laboratory data, vital signs, trends in data,31, 32 and newer proteomic markers (eg, procalcitonin).33 We also found that most transfers occurred early (within <72 hours), raising the possibility that at least some of these transfers may involve issues around triage rather than sudden deterioration.

Our study has important limitations. Due to resource constraints and limited data availability, we could not characterize the patients as well as might be desirable; in particular, we could not make full determinations of the actual reasons for patients' transfer for all patients. Broadly speaking, transfer to a higher level of care could be due to inappropriate triage, appropriate (preventive) transfer (which could include transfer to a more richly staffed unit for a specific procedure), relentless progression of disease despite maximal therapy, the occurrence of management errors, patient and family uncertainty about goals of care or inadequate understanding of treatment options and prognoses, or a combination of these factors. We could not make these distinctions with currently available electronic data. This is also true of postsurgical patients, in whom it is difficult to determine which transfers to intensive care might be planned (eg, in the case of surgical procedures where ICU care is anticipated) as opposed to the occurrence of a deterioration during or following surgery. Another major limitation of this study is our inability to identify code or no code status electronically. The elapsed LOS at time of death among patients who experienced a transfer to a higher level of care (as compared to patients who died in the ward without ever experiencing intra‐hospital transfer) suggests, but does not prove, that prolonged efforts were being made to keep them alive. We were also limited in terms of having access to other process data (eg, physician staffing levels, provision and timing of palliative care). Having ICU severity of illness scores would have permitted us to compare our cohort to those of other recent studies showing elevated mortality rates among transfer patients,911 but we have not yet developed that capability.

Consideration of our study findings suggests a possible research agenda that could be implemented by hospitalist researchers. This agenda should emphasize three areas: detection, intervention, and reflection.

With respect to detection, attention needs to be paid to better tools for quantifying patient risk at the time a decision to admit to the ward is made. It is likely that such tools will need to combine the attributes of our severity score (LAPS) with those of the manually assigned scores.30, 34 In some cases, use of these tools could lead a physician to change the locus of admission from the ward to the TCU or ICU, which could improve outcomes by ensuring more timely provision of intensive care. Since problems with initial triage could be due to factors other than the failure to suspect or anticipate impending instability, future research should also include a cognitive component (eg, quantifying what proportion of subsequent patient deteriorations could be ascribed to missed diagnoses35). Additional work also needs to be done on developing mathematical models that can inform electronic monitoring of ward (not just ICU) patients.

Research on interventions that hospitalists can use to prevent the need for intensive care or to improve the rescue rate should take two routes. The first is a disease‐specific route, which builds on the fact that a relatively small set of conditions (pneumonia, sepsis, gastrointestinal bleeding) account for most transfers to a higher level of care. Condition‐specific protocols, checklists, and bundles36 tailored to a ward environment (as opposed to the ICU or to the entire hospital) might prevent deteriorations in these patients, as has been reported for sepsis.37 The second route is to improve the overall capabilities of rapid response and code blue teams. Such research would need to include a more careful assessment of what commonalities exist among patients who were and were not successfully rescued by these teams. This approach would probably yield more insights than the current literature, which focuses on whether rapid response teams are a good thing or not.

Finally, research also needs to be performed on how hospitalists reflect on adverse outcomes among ward patients. Greater emphasis needs to be placed on moving beyond trigger tool approaches that rely on manual chart review. In an era of expanding use of electronic medical record systems, more work needs to be done on how to harness these to provide hospitalists with better quantitative and risk‐adjusted information. This information should not be limited to simply reporting rates of transfers and deaths. Rather, finer distinctions must be provided with respect of the type of patients (ie, more diagnostic detail), the clinical status of patients (ie, more physiologic detail), as well as the effects of including or excluding patients in whom therapeutic options may be limited (ie, do not resuscitate and comfort care patients) on reported rates. Ideally, researchers should develop better process and outcomes measures that could be tested in collaborative networks that include multiple nonacademic general medical‐surgical wards.

Considerable research and public attention is being paid to the quantification, risk adjustment, and reporting of inpatient mortality.15 Inpatient mortality is reported as aggregate mortality (for all hospitalized patients or those with a specific diagnosis3, 6) or intensive care unit (ICU) mortality.7, 8 While reporting aggregate hospital or aggregate ICU mortality rates is useful, it is also important to develop reporting strategies that go beyond simply using data elements found in administrative databases (eg, diagnosis and procedure codes) to quantify practice variation. Ideally, such strategies would permit delineating processes of careparticularly those potentially under the control of hospitalists, not only intensiviststo identify improvement opportunities. One such process, which can be tracked using the bed history component of a patient's electronic medical record, is the transfer of patients between different units within the same hospital.

Several studies have documented that risk of ICU death is highest among patients transferred from general medical‐surgical wards, intermediate among direct admissions from the emergency department, and lowest among surgical admissions.911 Opportunities to reduce subsequent ICU mortality have been studied among ward patients who develop sepsis and are then transferred to the ICU,12 among patients who experience cardiac arrest,13, 14 as well as among patients with any physiological deterioration (eg, through the use of rapid response teams).1517 Most of these studies have been single‐center studies and/or studies reporting only an ICU denominator. While useful in some respects, such studies are less helpful to hospitalists, who would benefit from better understanding of the types of patients transferred and the total impact that transfers to a higher level of care make on general medical‐surgical wards. In addition, entities such as the Institute for Healthcare Improvement recommend the manual review of records of patients who were transferred from the ward to the ICU18 to identify performance improvement opportunities. While laudable, such approaches do not lend themselves to automated reporting strategies.

We recently described a new risk adjustment methodology for inpatient mortality based entirely on automated data preceding hospital admission and not restricted to ICU patients. This methodology, which has been externally validated in Ottawa, Canada, after development in the Kaiser Permanente Medical Care Program (KPMCP), permits quantification of a patient's pre‐existing comorbidity burden, physiologic derangement at the time of admission, and overall inpatient mortality risk.19, 20 The primary purpose of this study was to combine this methodology with bed history analysis to quantify the in‐hospital mortality and length of stay (LOS) of patients who experienced intra‐hospital transfers in a large, multihospital system. As a secondary goal, we also wanted to assess the degree to which these transfers could be predicted based on information available prior to a patient's admission.

Materials and Methods

This project was approved by the Northern California KPMCP Institutional Review Board for the Protection of Human Subjects.

The Northern California KPMCP serves a total population of approximately 3.3 million members. Under a mutual exclusivity arrangement, physicians of The Permanente Medical Group, Inc., care for Kaiser Foundation Health Plan, Inc. members at facilities owned by Kaiser Foundation Hospitals, Inc. All Northern California KPMCP hospitals and clinics employ the same information systems with a common medical record number and can track care covered by the plan but delivered elsewhere. Databases maintained by the KPMCP capture admission and discharge times, admission and discharge diagnoses and procedures (assigned by professional coders), bed histories, inter‐hospital transfers, as well as the results of all inpatient and outpatient laboratory tests. The use of these databases for research has been described in multiple reports.2124

Our setting consisted of all 19 hospitals owned and operated by the KPMCP, whose characteristics are summarized in the Supporting Information Appendix available to interested readers. These include the 17 described in our previous report19 as well as 2 new hospitals (Antioch and Manteca) which are similar in size and type of population served. Our study population consisted of all patients admitted to these 19 hospitals who met these criteria: 1) hospitalization began from November 1st, 2006 through January 31st, 2008; 2) initial hospitalization occurred at a Northern California KPMCP hospital (ie, for inter‐hospital transfers, the first hospital stay occurred within the KPMCP); 3) age 15 years; and 4) hospitalization was not for childbirth.

We defined a linked hospitalization as the time period that began with a patient's admission to the hospital and ended with the patient's discharge (home, to a nursing home, or death). Linked hospitalizations can thus involve more than 1 hospital stay and could include a patient transfer from one hospital to another prior to definitive discharge. For linked hospitalizations, mortality was attributed to the admitting KPMCP hospital (ie, if a patient was admitted to hospital A, transferred to B, and died at hospital B, mortality was attributed to hospital A). We defined total LOS as the exact time in hours from when a patient was first admitted to the hospital until death or final discharge home or to a nursing home, while total ICU or transitional care unit (TCU, referred to as stepdown unit in some hospitals) LOS was calculated for all individual ICU or TCU stays during the hospital stay.

Results

During the study period, a total of 249,129 individual hospital stays involving 170,151 patients occurred at these 19 hospitals. After concatenation of inter‐hospital transfers, we were left with 237,208 linked hospitalizations. We excluded 26,738 linked hospitalizations that began at a non‐KPMCP hospital (ie, they were transported in), leaving a total of 210,470 linked hospitalizations involving 150,495 patients. The overall linked hospitalization mortality rate was 3.30%.

Table 1 summarizes cohort characteristics based on initial hospital location. On admission, ICU patients had the highest degree of physiologic derangement as well as the highest predicted mortality. Considerable inter‐hospital variation was present in both predictors and outcomes; details on these variations are provided in the Supporting Information Appendix.

Table 2 summarizes data from 3 groups of patients: patients initially admitted to the ward, or TCU, who did not experience a transfer to a higher level of care and patients admitted to these 2 units who did experience such a transfer. Patients who experienced a transfer constituted 5.3% (6,484/121,237) of ward patients and 6.7% (1,384/20,556) of TCU patients. Transferred patients tended to be older, have more acute physiologic derangement (higher LAPS), a greater pre‐existing illness burden (higher COPS), and a higher predicted mortality risk. Among ward patients, those with the following admission diagnoses were most likely to experience a transfer to a higher level of care: gastrointestinal bleeding (10.8% of all transfers), pneumonia (8.7%), and other infections (8.2%). The diagnoses most likely to be associated with death following transfer were cancer (death rate among transferred patients, 48%), renal disease (death rate, 36%), and liver disease (33%). Similar distributions were observed for TCU patients.

Table 3 compares outcomes among ward and TCU patients who did and did not experience a transfer to a higher level of care. The table shows that transferred patients were almost 3 times as likely to die, even after controlling for severity of illness, and that their hospital LOS was 9 days higher than expected. This increased risk was seen in all hospitals and among all transfer types (ward to ICU, ward to TCU, and TCU to ICU).

Table 3 also shows that, among decedent patients, those who never left the ward or TCU died much sooner than those who died following transfer. Among direct admits to the ICU, the median LOS at time of death was 3.9 days, with a mean of 9.4 standard deviation of 19.9 days, while the corresponding times for TCU direct admits were a median and mean LOS of 6.5 and 11.7 19.5 days.

Table 4 summarizes outcomes among different patient subgroups that did and did not experience a transfer to a higher level of care. Based on location, patients who experienced a transfer from the TCU to the ICU had the highest crude death rate, but patients transferred from the ward to the ICU had the highest OEMR. On the other hand, if one divides patients by the degree of physiologic derangement, patients with low LAPS who experienced a transfer had the highest OEMR. With respect to LOS, patients transferred from the TCU to the ICU had the highest OMELOS (13.4 extra days).

Transfers to a higher level of care at a different hospital, which in the KPMCP are usually planned, experienced lower mortality than transfers within the same hospital. For ward to TCU transfers, intra‐hospital transfers had a mortality of 12.1% while inter‐hospital transfers had a mortality of 5.7%. Corresponding rates for ward to ICU transfers were 21.7% and 11.2%, and for TCU to ICU transfers the rates were 25.9% and 12.5%, respectively.

Among patients initially admitted to the ward, a model to predict the occurrence of a transfer to a higher level of care (within 48 hours after admission) that included age, sex, admission type, primary condition, LAPS, COPS, and interaction terms had poor discrimination, with an area under the receiver operator characteristic (c statistic) of only 0.64. The c statistic for a model to predict transfer after 48 hours was 0.66. The corresponding models for TCU admits had c statistics of 0.67 and 0.68. All four models had poor calibration.

Discussion

Using automated bed history data permits characterizing a patient population with disproportionate mortality and LOS: intra‐hospital transfers to special care units (ICUs or TCUs). Indeed, the largest subset of these patients (those initially admitted to the ward or TCU) constituted only 3.7% of all admissions, but accounted for 24.2% of all ICU admissions, 21.7% of all hospital deaths, and 13.2% of all hospital days. These patients also had very elevated OEMRs and OMELOS. Models based on age, sex, preadmission laboratory test results, and comorbidities did not predict the occurrence of these transfers.

We performed multivariate analyses to explore the degree to which electronically assigned preadmission severity scores could predict these transfers. These analyses found that, compared to our ability to predict inpatient or 30‐day mortality at the time of admission, which is excellent, our ability to predict the occurrence of transfer after admission is much more limited. These results highlight the limitations of severity scores that rely on automated data, which may not have adequate discrimination when it comes to determining the risk of an adverse outcome within a narrow time frame. For example, among the 121,237 patients initially admitted to the ward who did not experience an intra‐hospital transfer, the mean LAPS was 18.9, while the mean LAPS among the 6,484 ward patients who did experience a transfer was 25.5. Differences between the mean and median LAPS, COPS, and predicted mortality risk among transferred and non‐transferred patients were significant (P < 0.0001 for all comparisons). However, examination of the distribution of LAPS, COPS, and predicted mortality risk between these two groups of patients showed considerable overlap.

Our methodology resembles Silber et al.'s27, 28 concept of failure to rescue in that it focuses on events occurring after hospitalization. Silber et al. argue that a hospital's quality can be measured by quantifying the degree to which patients who experience new problems are successfully rescued. Furthermore, quantification of those situations where rescue attempts are unsuccessful is felt to be superior to simply comparing raw or adjusted mortality rates because these are primarily determined by underlying case mix. The primary difference between Silber et al.'s approach and ours is at the level of detailthey specified a specific set of complications, whereas our measure is more generic and would include patients with many of the complications specified by Silber et al.27, 28

Most of the patients transferred to a higher level of care in our cohort survived (ie, were rescued), indicating that intensive care is beneficial. However, the fact that these patients had elevated OEMRs and OMELOS indicates that the real challenge facing hospitalists involves the timing of provision of a beneficial intervention. In theory, improved timing could result from earlier detection of problems, which is the underlying rationale for employing rapid response teams. However, the fact that our electronic tools (LAPS, COPS) cannot predict patient deteriorations within a narrow time frame suggests that early detection will remain a major challenge. Manually assigned vital signs scores designed for this purpose do not have good discrimination either.29, 30 This raises the possibility that, though patient groups may differ in terms of overall illness severity and mortality risk, differences at the individual patient level may be too subtle for clinicians to detect. Future research may thus need to focus on scores that combine laboratory data, vital signs, trends in data,31, 32 and newer proteomic markers (eg, procalcitonin).33 We also found that most transfers occurred early (within <72 hours), raising the possibility that at least some of these transfers may involve issues around triage rather than sudden deterioration.

Our study has important limitations. Due to resource constraints and limited data availability, we could not characterize the patients as well as might be desirable; in particular, we could not make full determinations of the actual reasons for patients' transfer for all patients. Broadly speaking, transfer to a higher level of care could be due to inappropriate triage, appropriate (preventive) transfer (which could include transfer to a more richly staffed unit for a specific procedure), relentless progression of disease despite maximal therapy, the occurrence of management errors, patient and family uncertainty about goals of care or inadequate understanding of treatment options and prognoses, or a combination of these factors. We could not make these distinctions with currently available electronic data. This is also true of postsurgical patients, in whom it is difficult to determine which transfers to intensive care might be planned (eg, in the case of surgical procedures where ICU care is anticipated) as opposed to the occurrence of a deterioration during or following surgery. Another major limitation of this study is our inability to identify code or no code status electronically. The elapsed LOS at time of death among patients who experienced a transfer to a higher level of care (as compared to patients who died in the ward without ever experiencing intra‐hospital transfer) suggests, but does not prove, that prolonged efforts were being made to keep them alive. We were also limited in terms of having access to other process data (eg, physician staffing levels, provision and timing of palliative care). Having ICU severity of illness scores would have permitted us to compare our cohort to those of other recent studies showing elevated mortality rates among transfer patients,911 but we have not yet developed that capability.

Consideration of our study findings suggests a possible research agenda that could be implemented by hospitalist researchers. This agenda should emphasize three areas: detection, intervention, and reflection.

With respect to detection, attention needs to be paid to better tools for quantifying patient risk at the time a decision to admit to the ward is made. It is likely that such tools will need to combine the attributes of our severity score (LAPS) with those of the manually assigned scores.30, 34 In some cases, use of these tools could lead a physician to change the locus of admission from the ward to the TCU or ICU, which could improve outcomes by ensuring more timely provision of intensive care. Since problems with initial triage could be due to factors other than the failure to suspect or anticipate impending instability, future research should also include a cognitive component (eg, quantifying what proportion of subsequent patient deteriorations could be ascribed to missed diagnoses35). Additional work also needs to be done on developing mathematical models that can inform electronic monitoring of ward (not just ICU) patients.

Research on interventions that hospitalists can use to prevent the need for intensive care or to improve the rescue rate should take two routes. The first is a disease‐specific route, which builds on the fact that a relatively small set of conditions (pneumonia, sepsis, gastrointestinal bleeding) account for most transfers to a higher level of care. Condition‐specific protocols, checklists, and bundles36 tailored to a ward environment (as opposed to the ICU or to the entire hospital) might prevent deteriorations in these patients, as has been reported for sepsis.37 The second route is to improve the overall capabilities of rapid response and code blue teams. Such research would need to include a more careful assessment of what commonalities exist among patients who were and were not successfully rescued by these teams. This approach would probably yield more insights than the current literature, which focuses on whether rapid response teams are a good thing or not.

Finally, research also needs to be performed on how hospitalists reflect on adverse outcomes among ward patients. Greater emphasis needs to be placed on moving beyond trigger tool approaches that rely on manual chart review. In an era of expanding use of electronic medical record systems, more work needs to be done on how to harness these to provide hospitalists with better quantitative and risk‐adjusted information. This information should not be limited to simply reporting rates of transfers and deaths. Rather, finer distinctions must be provided with respect of the type of patients (ie, more diagnostic detail), the clinical status of patients (ie, more physiologic detail), as well as the effects of including or excluding patients in whom therapeutic options may be limited (ie, do not resuscitate and comfort care patients) on reported rates. Ideally, researchers should develop better process and outcomes measures that could be tested in collaborative networks that include multiple nonacademic general medical‐surgical wards.

Considerable research and public attention is being paid to the quantification, risk adjustment, and reporting of inpatient mortality.15 Inpatient mortality is reported as aggregate mortality (for all hospitalized patients or those with a specific diagnosis3, 6) or intensive care unit (ICU) mortality.7, 8 While reporting aggregate hospital or aggregate ICU mortality rates is useful, it is also important to develop reporting strategies that go beyond simply using data elements found in administrative databases (eg, diagnosis and procedure codes) to quantify practice variation. Ideally, such strategies would permit delineating processes of careparticularly those potentially under the control of hospitalists, not only intensiviststo identify improvement opportunities. One such process, which can be tracked using the bed history component of a patient's electronic medical record, is the transfer of patients between different units within the same hospital.

Several studies have documented that risk of ICU death is highest among patients transferred from general medical‐surgical wards, intermediate among direct admissions from the emergency department, and lowest among surgical admissions.911 Opportunities to reduce subsequent ICU mortality have been studied among ward patients who develop sepsis and are then transferred to the ICU,12 among patients who experience cardiac arrest,13, 14 as well as among patients with any physiological deterioration (eg, through the use of rapid response teams).1517 Most of these studies have been single‐center studies and/or studies reporting only an ICU denominator. While useful in some respects, such studies are less helpful to hospitalists, who would benefit from better understanding of the types of patients transferred and the total impact that transfers to a higher level of care make on general medical‐surgical wards. In addition, entities such as the Institute for Healthcare Improvement recommend the manual review of records of patients who were transferred from the ward to the ICU18 to identify performance improvement opportunities. While laudable, such approaches do not lend themselves to automated reporting strategies.

We recently described a new risk adjustment methodology for inpatient mortality based entirely on automated data preceding hospital admission and not restricted to ICU patients. This methodology, which has been externally validated in Ottawa, Canada, after development in the Kaiser Permanente Medical Care Program (KPMCP), permits quantification of a patient's pre‐existing comorbidity burden, physiologic derangement at the time of admission, and overall inpatient mortality risk.19, 20 The primary purpose of this study was to combine this methodology with bed history analysis to quantify the in‐hospital mortality and length of stay (LOS) of patients who experienced intra‐hospital transfers in a large, multihospital system. As a secondary goal, we also wanted to assess the degree to which these transfers could be predicted based on information available prior to a patient's admission.

Materials and Methods

This project was approved by the Northern California KPMCP Institutional Review Board for the Protection of Human Subjects.

The Northern California KPMCP serves a total population of approximately 3.3 million members. Under a mutual exclusivity arrangement, physicians of The Permanente Medical Group, Inc., care for Kaiser Foundation Health Plan, Inc. members at facilities owned by Kaiser Foundation Hospitals, Inc. All Northern California KPMCP hospitals and clinics employ the same information systems with a common medical record number and can track care covered by the plan but delivered elsewhere. Databases maintained by the KPMCP capture admission and discharge times, admission and discharge diagnoses and procedures (assigned by professional coders), bed histories, inter‐hospital transfers, as well as the results of all inpatient and outpatient laboratory tests. The use of these databases for research has been described in multiple reports.2124

Our setting consisted of all 19 hospitals owned and operated by the KPMCP, whose characteristics are summarized in the Supporting Information Appendix available to interested readers. These include the 17 described in our previous report19 as well as 2 new hospitals (Antioch and Manteca) which are similar in size and type of population served. Our study population consisted of all patients admitted to these 19 hospitals who met these criteria: 1) hospitalization began from November 1st, 2006 through January 31st, 2008; 2) initial hospitalization occurred at a Northern California KPMCP hospital (ie, for inter‐hospital transfers, the first hospital stay occurred within the KPMCP); 3) age 15 years; and 4) hospitalization was not for childbirth.

We defined a linked hospitalization as the time period that began with a patient's admission to the hospital and ended with the patient's discharge (home, to a nursing home, or death). Linked hospitalizations can thus involve more than 1 hospital stay and could include a patient transfer from one hospital to another prior to definitive discharge. For linked hospitalizations, mortality was attributed to the admitting KPMCP hospital (ie, if a patient was admitted to hospital A, transferred to B, and died at hospital B, mortality was attributed to hospital A). We defined total LOS as the exact time in hours from when a patient was first admitted to the hospital until death or final discharge home or to a nursing home, while total ICU or transitional care unit (TCU, referred to as stepdown unit in some hospitals) LOS was calculated for all individual ICU or TCU stays during the hospital stay.

Results

During the study period, a total of 249,129 individual hospital stays involving 170,151 patients occurred at these 19 hospitals. After concatenation of inter‐hospital transfers, we were left with 237,208 linked hospitalizations. We excluded 26,738 linked hospitalizations that began at a non‐KPMCP hospital (ie, they were transported in), leaving a total of 210,470 linked hospitalizations involving 150,495 patients. The overall linked hospitalization mortality rate was 3.30%.

Table 1 summarizes cohort characteristics based on initial hospital location. On admission, ICU patients had the highest degree of physiologic derangement as well as the highest predicted mortality. Considerable inter‐hospital variation was present in both predictors and outcomes; details on these variations are provided in the Supporting Information Appendix.

Table 2 summarizes data from 3 groups of patients: patients initially admitted to the ward, or TCU, who did not experience a transfer to a higher level of care and patients admitted to these 2 units who did experience such a transfer. Patients who experienced a transfer constituted 5.3% (6,484/121,237) of ward patients and 6.7% (1,384/20,556) of TCU patients. Transferred patients tended to be older, have more acute physiologic derangement (higher LAPS), a greater pre‐existing illness burden (higher COPS), and a higher predicted mortality risk. Among ward patients, those with the following admission diagnoses were most likely to experience a transfer to a higher level of care: gastrointestinal bleeding (10.8% of all transfers), pneumonia (8.7%), and other infections (8.2%). The diagnoses most likely to be associated with death following transfer were cancer (death rate among transferred patients, 48%), renal disease (death rate, 36%), and liver disease (33%). Similar distributions were observed for TCU patients.

Table 3 compares outcomes among ward and TCU patients who did and did not experience a transfer to a higher level of care. The table shows that transferred patients were almost 3 times as likely to die, even after controlling for severity of illness, and that their hospital LOS was 9 days higher than expected. This increased risk was seen in all hospitals and among all transfer types (ward to ICU, ward to TCU, and TCU to ICU).

Table 3 also shows that, among decedent patients, those who never left the ward or TCU died much sooner than those who died following transfer. Among direct admits to the ICU, the median LOS at time of death was 3.9 days, with a mean of 9.4 standard deviation of 19.9 days, while the corresponding times for TCU direct admits were a median and mean LOS of 6.5 and 11.7 19.5 days.

Table 4 summarizes outcomes among different patient subgroups that did and did not experience a transfer to a higher level of care. Based on location, patients who experienced a transfer from the TCU to the ICU had the highest crude death rate, but patients transferred from the ward to the ICU had the highest OEMR. On the other hand, if one divides patients by the degree of physiologic derangement, patients with low LAPS who experienced a transfer had the highest OEMR. With respect to LOS, patients transferred from the TCU to the ICU had the highest OMELOS (13.4 extra days).

Transfers to a higher level of care at a different hospital, which in the KPMCP are usually planned, experienced lower mortality than transfers within the same hospital. For ward to TCU transfers, intra‐hospital transfers had a mortality of 12.1% while inter‐hospital transfers had a mortality of 5.7%. Corresponding rates for ward to ICU transfers were 21.7% and 11.2%, and for TCU to ICU transfers the rates were 25.9% and 12.5%, respectively.

Among patients initially admitted to the ward, a model to predict the occurrence of a transfer to a higher level of care (within 48 hours after admission) that included age, sex, admission type, primary condition, LAPS, COPS, and interaction terms had poor discrimination, with an area under the receiver operator characteristic (c statistic) of only 0.64. The c statistic for a model to predict transfer after 48 hours was 0.66. The corresponding models for TCU admits had c statistics of 0.67 and 0.68. All four models had poor calibration.

Discussion

Using automated bed history data permits characterizing a patient population with disproportionate mortality and LOS: intra‐hospital transfers to special care units (ICUs or TCUs). Indeed, the largest subset of these patients (those initially admitted to the ward or TCU) constituted only 3.7% of all admissions, but accounted for 24.2% of all ICU admissions, 21.7% of all hospital deaths, and 13.2% of all hospital days. These patients also had very elevated OEMRs and OMELOS. Models based on age, sex, preadmission laboratory test results, and comorbidities did not predict the occurrence of these transfers.

We performed multivariate analyses to explore the degree to which electronically assigned preadmission severity scores could predict these transfers. These analyses found that, compared to our ability to predict inpatient or 30‐day mortality at the time of admission, which is excellent, our ability to predict the occurrence of transfer after admission is much more limited. These results highlight the limitations of severity scores that rely on automated data, which may not have adequate discrimination when it comes to determining the risk of an adverse outcome within a narrow time frame. For example, among the 121,237 patients initially admitted to the ward who did not experience an intra‐hospital transfer, the mean LAPS was 18.9, while the mean LAPS among the 6,484 ward patients who did experience a transfer was 25.5. Differences between the mean and median LAPS, COPS, and predicted mortality risk among transferred and non‐transferred patients were significant (P < 0.0001 for all comparisons). However, examination of the distribution of LAPS, COPS, and predicted mortality risk between these two groups of patients showed considerable overlap.

Our methodology resembles Silber et al.'s27, 28 concept of failure to rescue in that it focuses on events occurring after hospitalization. Silber et al. argue that a hospital's quality can be measured by quantifying the degree to which patients who experience new problems are successfully rescued. Furthermore, quantification of those situations where rescue attempts are unsuccessful is felt to be superior to simply comparing raw or adjusted mortality rates because these are primarily determined by underlying case mix. The primary difference between Silber et al.'s approach and ours is at the level of detailthey specified a specific set of complications, whereas our measure is more generic and would include patients with many of the complications specified by Silber et al.27, 28

Most of the patients transferred to a higher level of care in our cohort survived (ie, were rescued), indicating that intensive care is beneficial. However, the fact that these patients had elevated OEMRs and OMELOS indicates that the real challenge facing hospitalists involves the timing of provision of a beneficial intervention. In theory, improved timing could result from earlier detection of problems, which is the underlying rationale for employing rapid response teams. However, the fact that our electronic tools (LAPS, COPS) cannot predict patient deteriorations within a narrow time frame suggests that early detection will remain a major challenge. Manually assigned vital signs scores designed for this purpose do not have good discrimination either.29, 30 This raises the possibility that, though patient groups may differ in terms of overall illness severity and mortality risk, differences at the individual patient level may be too subtle for clinicians to detect. Future research may thus need to focus on scores that combine laboratory data, vital signs, trends in data,31, 32 and newer proteomic markers (eg, procalcitonin).33 We also found that most transfers occurred early (within <72 hours), raising the possibility that at least some of these transfers may involve issues around triage rather than sudden deterioration.

Our study has important limitations. Due to resource constraints and limited data availability, we could not characterize the patients as well as might be desirable; in particular, we could not make full determinations of the actual reasons for patients' transfer for all patients. Broadly speaking, transfer to a higher level of care could be due to inappropriate triage, appropriate (preventive) transfer (which could include transfer to a more richly staffed unit for a specific procedure), relentless progression of disease despite maximal therapy, the occurrence of management errors, patient and family uncertainty about goals of care or inadequate understanding of treatment options and prognoses, or a combination of these factors. We could not make these distinctions with currently available electronic data. This is also true of postsurgical patients, in whom it is difficult to determine which transfers to intensive care might be planned (eg, in the case of surgical procedures where ICU care is anticipated) as opposed to the occurrence of a deterioration during or following surgery. Another major limitation of this study is our inability to identify code or no code status electronically. The elapsed LOS at time of death among patients who experienced a transfer to a higher level of care (as compared to patients who died in the ward without ever experiencing intra‐hospital transfer) suggests, but does not prove, that prolonged efforts were being made to keep them alive. We were also limited in terms of having access to other process data (eg, physician staffing levels, provision and timing of palliative care). Having ICU severity of illness scores would have permitted us to compare our cohort to those of other recent studies showing elevated mortality rates among transfer patients,911 but we have not yet developed that capability.

Consideration of our study findings suggests a possible research agenda that could be implemented by hospitalist researchers. This agenda should emphasize three areas: detection, intervention, and reflection.

With respect to detection, attention needs to be paid to better tools for quantifying patient risk at the time a decision to admit to the ward is made. It is likely that such tools will need to combine the attributes of our severity score (LAPS) with those of the manually assigned scores.30, 34 In some cases, use of these tools could lead a physician to change the locus of admission from the ward to the TCU or ICU, which could improve outcomes by ensuring more timely provision of intensive care. Since problems with initial triage could be due to factors other than the failure to suspect or anticipate impending instability, future research should also include a cognitive component (eg, quantifying what proportion of subsequent patient deteriorations could be ascribed to missed diagnoses35). Additional work also needs to be done on developing mathematical models that can inform electronic monitoring of ward (not just ICU) patients.

Research on interventions that hospitalists can use to prevent the need for intensive care or to improve the rescue rate should take two routes. The first is a disease‐specific route, which builds on the fact that a relatively small set of conditions (pneumonia, sepsis, gastrointestinal bleeding) account for most transfers to a higher level of care. Condition‐specific protocols, checklists, and bundles36 tailored to a ward environment (as opposed to the ICU or to the entire hospital) might prevent deteriorations in these patients, as has been reported for sepsis.37 The second route is to improve the overall capabilities of rapid response and code blue teams. Such research would need to include a more careful assessment of what commonalities exist among patients who were and were not successfully rescued by these teams. This approach would probably yield more insights than the current literature, which focuses on whether rapid response teams are a good thing or not.

Finally, research also needs to be performed on how hospitalists reflect on adverse outcomes among ward patients. Greater emphasis needs to be placed on moving beyond trigger tool approaches that rely on manual chart review. In an era of expanding use of electronic medical record systems, more work needs to be done on how to harness these to provide hospitalists with better quantitative and risk‐adjusted information. This information should not be limited to simply reporting rates of transfers and deaths. Rather, finer distinctions must be provided with respect of the type of patients (ie, more diagnostic detail), the clinical status of patients (ie, more physiologic detail), as well as the effects of including or excluding patients in whom therapeutic options may be limited (ie, do not resuscitate and comfort care patients) on reported rates. Ideally, researchers should develop better process and outcomes measures that could be tested in collaborative networks that include multiple nonacademic general medical‐surgical wards.