User login
Patients at risk for clinical deterioration in the inpatient setting may not be identified efficiently or effectively by health care providers. Early warning systems that link clinical observations to rapid response mechanisms (such as medical emergency teams) have the potential to improve outcomes, but rigorous studies are lacking.1 The pediatric Rothman Index (pRI) is an automated early warning system sold by the company PeraHealth that is integrated with the electronic health record. The system incorporates vital signs, labs, and nursing assessments from existing electronic health record data to provide a single numeric score that generates alerts based on low absolute scores and acute decreases in score (low scores indicate high mortality risk).2 Automated alerts or rules based on the pRI score are meant to bring important changes in clinical status to the attention of clinicians.
Adverse outcomes (eg, unplanned intensive care unit [ICU] transfers and mortality) are associated with low pRI scores, and scores appear to decline prior to such events.2 However, the limitation of this and other studies evaluating the sensitivity of early warning systems3-6 is that the generated alerts are assigned “true positive” status if they precede clinical deterioration, regardless of whether or not they provide meaningful information to the clinicians caring for the patients. There are two potential critiques of this approach. First, the alert may have preceded a deterioration event but may not have been clinically relevant (eg, an alert triggered by a finding unrelated to the patient’s acute health status, such as a scar that was newly documented as an abnormal skin finding and as a result led to a worsening in the pRI). Second, even if the preceding alert demonstrated clinical relevance to a deterioration event, the clinicians at the bedside may have been aware of the patient’s deterioration for hours and have already escalated care. In this situation, the alert would simply confirm what the clinician already knew.
To better understand the relationship between early warning system acuity alerts and clinical practice, we examined a cohort of hospitalized patients who experienced a critical deterioration event (CDE)7 and who would have triggered a preceding pRI alert. We evaluated the clinical relationship of the alert to the CDE (ie, whether the alert reflected physiologic changes related to a CDE or was instead an artifact of documentation) and identified whether the alert would have preceded evidence that clinicians recognized deterioration or escalated care.
METHODS
Patients and Setting
This retrospective cross-sectional study was performed at Children’s Hospital of Philadelphia (CHOP), a freestanding children’s hospital with 546 beds. Eligible patients were hospitalized on nonintensive care, noncardiology, surgical wards between January 1, 2013, and December 31, 2013. The CHOP Institutional Review Board (IRB) approved the study with waivers of consent and assent. A HIPAA Business Associate Agreement and an IRB Reliance Agreement were in place with PeraHealth to permit data transfer.
Definition of Critical Deterioration Events
Critical deterioration events (CDEs) were defined according to an existing, validated measure7 as unplanned transfers to the ICU with continuous or bilevel positive airway pressure, tracheal intubation, and/or vasopressor infusion in the 12 hours after transfer. At CHOP, all unplanned ICU transfers are routed through the hospital’s rapid response or code blue teams, so these patients were identified using an existing database managed by the CHOP Resuscitation Committee. In the database, the elements of CDEs are entered as part of ongoing quality improvement activities. The time of CDE was defined as the time of the rapid response call precipitating unplanned transfer to the ICU.
The Pediatric Rothman Index
The pRI is an automated acuity score that has been validated in hospitalized pediatric patients.2 The pRI is calculated using existing variables from the electronic health record, including manually entered vital signs, laboratory values, cardiac rhythm, and nursing assessments of organ systems. The weights assigned to continuous variables are a function of deviation from the norm.2,8 (See Supplement 1 for a complete list of variables.)
The pRI is integrated with the electronic health record and automatically generates a score each time a new data observation becomes available. Changes in score over time and low absolute scores generate a graduated series of alerts ranging from medium to very high acuity. This analysis used PeraHealth’s standard pRI alerts. Medium acuity alerts occurred when the pRI score decreased by ≥30% in 24 hours. A high acuity alert occurred when the pRI score decreased by ≥40% in 6 hours. A very high acuity alert occurred when the pRI absolute score was ≤ 30.
Development of the Source Dataset
In 2014, CHOP shared one year of clinical data with PeraHealth as part of the process of deciding whether or not to implement the pRI. The pRI algorithm retrospectively generated scores and acuity alerts for all CHOP patients who experienced CDEs between January 1, 2013, and December 31, 2013. The pRI algorithm was not active in the hospital environment during this time period; the scores and acuity alerts were not visible to clinicians. This dataset was provided to the investigators at CHOP to conduct this project.
Data Collection
Pediatric intensive care nurses trained in clinical research data abstraction from the CHOP Critical Care Center for Evidence and Outcomes performed the chart review for this study. Chart abstraction comparisons were completed on the first 15 charts to ensure interrater reliability, and additional quality assurance checks were performed on intermittent charts to ensure consistency and definition adherence. We managed all data using Research Electronic Data Capture.9
To study the value of alerts labeled as “true positives,” we restricted the dataset to CDEs in which acuity alert(s) within the prior 72 hours would have been triggered if the pRI had been in clinical use at the time.
To identify the clinical relationship between pRI and CDE, we reviewed each chart with the goal of determining whether the preceding acuity alerts were clinically associated with the etiology of the CDE. We determined the etiology of the CDE by reviewing the cause(s) identified in the note written by rapid response or code blue team responders or by the admitting clinical team after transfer to the ICU. We then used a tool provided by PeraHealth to identify the specific score components that led to worsening pRI. If the score components that worsened were (a) consistent with a clinical change as opposed to a documentation artifact and (b) an organ system change that was plausibly related to the CDE etiology, we concluded that the alert was clinically related to the etiology of the CDE.
We defined documentation artifacts as instances in nursing documentation in which a finding unrelated to the patient’s acute health status, such as a scar, was newly documented as abnormal and led to worsening pRI. Any cases in which the clinical relevance was unclear underwent review by additional members of the team
To determine the temporal relationship among pRI, CDE, and clinician awareness or action, we then sought to systematically determine whether the preceding acuity alerts preceded documented evidence of clinicians recognizing deterioration or escalation of care. We made the a priori decision that acuity alerts that occurred more than 24 hours prior to a deterioration event had questionable clinical actionability. Therefore, we restricted this next analysis to CDEs with acuity alerts during the 24 hours prior to a CDE. We reviewed time-stamped progress notes written by clinicians in the 24 hours period prior to the time of the CDE and identified whether the notes reflected an adverse change in patient status or a clinical intervention. We then compared the times of these notes with the times of the alerts and CDEs. Given that documentation of change in clinical status often occurs after clinical intervention, we also reviewed new orders placed in the 24 hours prior to each CDE to determine escalation of care. We identified the following orders as reflective of escalation of care independent of specific disease process: administration of intravenous fluid bolus, blood product, steroid, or antibiotic, increased respiratory support, new imaging studies, and new laboratory studies. We then compared the time of each order with the time of the alert and CDE.
RESULTS
During the study period, 73 events met the CDE criteria and had a pRI alert during admission. Of the 73 events, 50 would have triggered at least one pRI alert in the 72-hour period leading up to the CDE (sensitivity 68%). Of the 50 events, 39 generated pRI alerts in the 24 hours leading up to the event, and 11 others generated pRI alerts between 24 and 72 hours prior to the event but did not generate any alerts during the 24 hours leading up to the event (Figure).
Patient Characteristics
The 50 CDEs labeled as true positives occurred in 46 unique patients. Table 1 displays the event characteristics.
Acuity Alerts
A total of 79 pRI alerts preceded the 50 CDEs. Of these acuity alerts, 44 (56%) were medium acuity alerts, 17 (22%) were high acuity alerts, and 18 (23%) were very high acuity alerts. Of the 50 CDEs that would have triggered pRI alerts, 33 (66%) would have triggered a single acuity alert and 17 (34%) would have triggered multiple acuity alerts.
Of the 50 CDEs, 39 (78%) had a preceding acuity alert within 24 hours prior to the CDE. In these cases, the alert preceded the CDE by a median of 3.1 hours (interquartile range of 0.7 to 10.3 hours).
We assessed the score components that caused each alert to trigger. All of the vital sign and laboratory components were assessed as clinically related to the CDE’s etiology. By contrast, about half of nursing assessment components were assessed as clinically related to the etiology of the CDE (Table 2). Abnormal cardiac, respiratory, and neurologic assessments were most frequently assessed as clinically relevant.
Escalation Orders
To determine whether the pRI alert would have preceded the earliest documented treatment efforts, we restricted evaluation to the 39 CDEs that had at least one alert in the 24-hour window prior to the CDE. When we reviewed escalation orders placed by clinicians, we found that in 26 cases (67%), the first clinician order reflecting escalation of care would have preceded the first pRI alert within the 24-hour period prior to the CDE. In 13 cases (33%), the first pRI alert would have preceded the first escalation order placed by the clinician. The first pRI alert and the first escalation order would have occurred within the same 1-hour period in 6 of these cases.
Provider Notes
Temporal Relationships
In Supplement 2, we present the proportion of CDEs in which the order or note preceded the pRI alert for each abnormal organ system.
The Figure shows the temporal relationships among escalation orders, clinician notes, and acuity alerts for the 39 CDEs with one or more alerts in the 24 hours leading up to the event. In 21 cases (54%), both an escalation order and a note preceded the first acuity alert. In 14 cases (36%), either an escalation order or a note preceded the first acuity alert. In four cases (10%), the alert preceded any documented evidence that clinicians had recognized deterioration or escalating care.
DISCUSSION
The main finding of this study is that 90% of CDE events that generated “true positive” pRI alerts had evidence suggesting that clinicians had already recognized deterioration and/or were already escalating care before most pRI alerts would have been triggered.
The impacts of early warning scores on patient safety outcomes are not well established. In a recent 21-hospital cluster randomized trial of the BedsidePEWS, a pediatric early warning score system, investigators found that implementing the system does not significantly decrease all-cause mortality in hospitalized children, although hospitals using the BedsidePEWS have low rates of significant CDEs.10 In other studies, early warning scores were often coimplemented with rapid response teams, and separating the incremental benefit of the scoring tool from the availability of a rapid response team is usually not possible.11
Therefore, the benefits of early warning scores are often inferred based on their test characteristics (eg, sensitivity and positive predictive value).12 Sensitivity, which is the proportion of patients who deteriorated and also triggered the early warning score within a reasonable time window preceding the event, is an important consideration when deciding whether an early warning score is worth implementing. A challenging follow-up question that goes beyond sensitivity is how often an early warning score adds new knowledge by identifying patients on a path toward deterioration who were not yet recognized. This study is the first to address that follow-up question. Our results revealed that the score appeared to precede evidence of clinician recognition of deterioration in 10% of CDEs. In some patients, the alert could have contributed to a detection of deterioration that was not previously evident. In the portion of CDEs in which the alert and escalation order or note occurred within the same one-hour window, the alert could have been used as confirmation of clinical suspicion. Notably, we did not evaluate the 16 cases in which a CDE preceded any pRI alert because we chose to focus on “true positive” cases in which pRI alerts preceded CDEs. These events could have had timely recognition by clinicians that we did not capture, so these results may provide an overestimation of CDEs in which the pRI preceded clinician recognition.
Prior work has described a range of mechanisms by which early warning scores can impact patient safety.13 The results of this study suggest limited incremental benefit for the pRI to alert physicians and nurses to new concerning changes at this hospital, although the benefits to low-resourced community hospitals that care for children may be great. The pRI score may also serve as evidence that empowers nurses to overcome barriers to further escalate care, even if the process of escalation has already begun. In addition to empowering nurses, the score may support trainees and clinicians with varying levels of pediatric expertise in the decision to escalate care. Evaluating these potential benefits would require prospective study.
We used the pRI alerts as they were already defined by PeraHealth for CHOP, and different alert thresholds may change score performance. Our study did not identify additional variables to improve score performance, but they can be investigated in future research.
This study had several limitations. First, this work is a single-center study with highly skilled pediatric providers, a mature rapid response system, and low rates of cardiopulmonary arrest outside ICUs. Therefore, the results that we obtained were not immediately generalizable. In a community environment with nurses and physicians who are less experienced in caring for ill children, an early warning score with high sensitivity may be beneficial in ensuring patient safety.
Second, by using escalation orders and notes from the patient chart, we did not capture all the undocumented ways in which clinicians demonstrate awareness of deterioration. For example, a resident may alert the attending on service or a team may informally request consultation with a specialist. We also gave equal weight to escalation orders and clinician notes as evidence of recognition of deterioration. It could be that either orders or notes more closely correlated with clinician awareness.
Finally, the data were from 2013. Although the score components have not changed, efforts to standardize nursing assessments may have altered the performance of the score in the intervening years.
CONCLUSIONS
In most patients who had a CDE at a large freestanding children’s hospital, escalation orders or documented changes in patient status would have occurred before a pRI alert. However, in a minority of patients, the alert could have contributed to the detection of deterioration that was not previously evident.
Disclosures
The authors have nothing to disclose
Funding
The study was supported by funds from the Department of Biomedical and Health Informatics at Children’s Hospital of Philadelphia. PeraHealth, the company that sells the Rothman Index software, provided a service to the investigators but no funding. They applied their proprietary scoring algorithm to the data from Children’s Hospital of Philadelphia to generate alerts retrospectively. This service was provided free of charge in 2014 during the time period when Children’s Hospital of Philadelphia was considering purchasing and implementing PeraHealth software, which it subsequently did. We did not receive any funding for the study from PeraHealth. PeraHealth personnel did not influence the study design, the interpretation of data, the writing of the report, or the decision to submit the article for publication.
1. Alam N, Hobbelink EL, van Tienhoven AJ, van de Ven PM, Jansma EP, Nanayakkara PWB. The impact of the use of the Early Warning Score (EWS) on patient outcomes: a systematic review. Resuscitation. 2014;85(5):587-594. doi: 10.1016/j.resuscitation.2014.01.013. PubMed
2. Rothman MJ, Tepas JJ, Nowalk AJ, et al. Development and validation of a continuously age-adjusted measure of patient condition for hospitalized children using the electronic medical record. J Biomed Inform. 2017;66 (Supplement C):180-193. doi: 10.1016/j.jbi.2016.12.013. PubMed
3. Akre M, Finkelstein M, Erickson M, Liu M, Vanderbilt L, Billman G. Sensitivity of the pediatric early warning score to identify patient deterioration. Pediatrics. 2010;125(4):e763-e769. doi: 10.1542/peds.2009-0338. PubMed
4. Seiger N, Maconochie I, Oostenbrink R, Moll HA. Validity of different pediatric early warning scores in the emergency department. Pediatrics. 2013;132(4):e841-e850. doi: 10.1542/peds.2012-3594. PubMed
5. Parshuram CS, Hutchison J, Middaugh K. Development and initial validation of the Bedside Paediatric Early Warning System score. Crit Care Lond Engl. 2009;13(4):R135. doi: 10.1186/cc7998. PubMed
6. Hollis RH, Graham LA, Lazenby JP, et al. A role for the early warning score in early identification of critical postoperative complications. Ann Surg. 2016;263(5):918-923. doi: 10.1097/SLA.0000000000001514. PubMed
7. Bonafide CP, Roberts KE, Priestley MA, et al. Development of a pragmatic measure for evaluating and optimizing rapid response systems. Pediatrics. 2012;129(4):e874-e881. doi: 10.1542/peds.2011-2784. PubMed
8. Rothman MJ, Rothman SI, Beals J. Development and validation of a continuous measure of patient condition using the electronic medical record. J Biomed Inform. 2013;46(5):837-848. doi: 10.1016/j.jbi.2013.06.011. PubMed
9. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research Electronic Data Capture (REDCap) - A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377-381. doi: 10.1016/j.jbi.2008.08.010. PubMed
10. Parshuram CS, Dryden-Palmer K, Farrell C, et al. Effect of a pediatric early warning system on all-cause mortality in hospitalized pediatric patients: the EPOCH randomized clinical trial. JAMA. 2018;319(10):1002-1012. doi: 10.1001/jama.2018.0948. PubMed
11. Bonafide CP, Localio AR, Roberts KE, Nadkarni VM, Weirich CM, Keren R. Impact of rapid response system implementation on critical deterioration events in children. JAMA Pediatr. 2014;168(1):25-33. doi: 10.1001/jamapediatrics.2013.3266. PubMed
12. Romero-Brufau S, Huddleston JM, Escobar GJ, Liebow M. Why the C-statistic is not informative to evaluate early warning scores and what metrics to use. Crit Care. 2015;19:285. doi: 10.1186/s13054-015-0999-1. PubMed
13. Bonafide CP, Roberts KE, Weirich CM, et al. Beyond statistical prediction: qualitative evaluation of the mechanisms by which pediatric early warning scores impact patient safety. J Hosp Med. 2013;8(5):248-253. doi: 10.1002/jhm.2026. PubMed
Patients at risk for clinical deterioration in the inpatient setting may not be identified efficiently or effectively by health care providers. Early warning systems that link clinical observations to rapid response mechanisms (such as medical emergency teams) have the potential to improve outcomes, but rigorous studies are lacking.1 The pediatric Rothman Index (pRI) is an automated early warning system sold by the company PeraHealth that is integrated with the electronic health record. The system incorporates vital signs, labs, and nursing assessments from existing electronic health record data to provide a single numeric score that generates alerts based on low absolute scores and acute decreases in score (low scores indicate high mortality risk).2 Automated alerts or rules based on the pRI score are meant to bring important changes in clinical status to the attention of clinicians.
Adverse outcomes (eg, unplanned intensive care unit [ICU] transfers and mortality) are associated with low pRI scores, and scores appear to decline prior to such events.2 However, the limitation of this and other studies evaluating the sensitivity of early warning systems3-6 is that the generated alerts are assigned “true positive” status if they precede clinical deterioration, regardless of whether or not they provide meaningful information to the clinicians caring for the patients. There are two potential critiques of this approach. First, the alert may have preceded a deterioration event but may not have been clinically relevant (eg, an alert triggered by a finding unrelated to the patient’s acute health status, such as a scar that was newly documented as an abnormal skin finding and as a result led to a worsening in the pRI). Second, even if the preceding alert demonstrated clinical relevance to a deterioration event, the clinicians at the bedside may have been aware of the patient’s deterioration for hours and have already escalated care. In this situation, the alert would simply confirm what the clinician already knew.
To better understand the relationship between early warning system acuity alerts and clinical practice, we examined a cohort of hospitalized patients who experienced a critical deterioration event (CDE)7 and who would have triggered a preceding pRI alert. We evaluated the clinical relationship of the alert to the CDE (ie, whether the alert reflected physiologic changes related to a CDE or was instead an artifact of documentation) and identified whether the alert would have preceded evidence that clinicians recognized deterioration or escalated care.
METHODS
Patients and Setting
This retrospective cross-sectional study was performed at Children’s Hospital of Philadelphia (CHOP), a freestanding children’s hospital with 546 beds. Eligible patients were hospitalized on nonintensive care, noncardiology, surgical wards between January 1, 2013, and December 31, 2013. The CHOP Institutional Review Board (IRB) approved the study with waivers of consent and assent. A HIPAA Business Associate Agreement and an IRB Reliance Agreement were in place with PeraHealth to permit data transfer.
Definition of Critical Deterioration Events
Critical deterioration events (CDEs) were defined according to an existing, validated measure7 as unplanned transfers to the ICU with continuous or bilevel positive airway pressure, tracheal intubation, and/or vasopressor infusion in the 12 hours after transfer. At CHOP, all unplanned ICU transfers are routed through the hospital’s rapid response or code blue teams, so these patients were identified using an existing database managed by the CHOP Resuscitation Committee. In the database, the elements of CDEs are entered as part of ongoing quality improvement activities. The time of CDE was defined as the time of the rapid response call precipitating unplanned transfer to the ICU.
The Pediatric Rothman Index
The pRI is an automated acuity score that has been validated in hospitalized pediatric patients.2 The pRI is calculated using existing variables from the electronic health record, including manually entered vital signs, laboratory values, cardiac rhythm, and nursing assessments of organ systems. The weights assigned to continuous variables are a function of deviation from the norm.2,8 (See Supplement 1 for a complete list of variables.)
The pRI is integrated with the electronic health record and automatically generates a score each time a new data observation becomes available. Changes in score over time and low absolute scores generate a graduated series of alerts ranging from medium to very high acuity. This analysis used PeraHealth’s standard pRI alerts. Medium acuity alerts occurred when the pRI score decreased by ≥30% in 24 hours. A high acuity alert occurred when the pRI score decreased by ≥40% in 6 hours. A very high acuity alert occurred when the pRI absolute score was ≤ 30.
Development of the Source Dataset
In 2014, CHOP shared one year of clinical data with PeraHealth as part of the process of deciding whether or not to implement the pRI. The pRI algorithm retrospectively generated scores and acuity alerts for all CHOP patients who experienced CDEs between January 1, 2013, and December 31, 2013. The pRI algorithm was not active in the hospital environment during this time period; the scores and acuity alerts were not visible to clinicians. This dataset was provided to the investigators at CHOP to conduct this project.
Data Collection
Pediatric intensive care nurses trained in clinical research data abstraction from the CHOP Critical Care Center for Evidence and Outcomes performed the chart review for this study. Chart abstraction comparisons were completed on the first 15 charts to ensure interrater reliability, and additional quality assurance checks were performed on intermittent charts to ensure consistency and definition adherence. We managed all data using Research Electronic Data Capture.9
To study the value of alerts labeled as “true positives,” we restricted the dataset to CDEs in which acuity alert(s) within the prior 72 hours would have been triggered if the pRI had been in clinical use at the time.
To identify the clinical relationship between pRI and CDE, we reviewed each chart with the goal of determining whether the preceding acuity alerts were clinically associated with the etiology of the CDE. We determined the etiology of the CDE by reviewing the cause(s) identified in the note written by rapid response or code blue team responders or by the admitting clinical team after transfer to the ICU. We then used a tool provided by PeraHealth to identify the specific score components that led to worsening pRI. If the score components that worsened were (a) consistent with a clinical change as opposed to a documentation artifact and (b) an organ system change that was plausibly related to the CDE etiology, we concluded that the alert was clinically related to the etiology of the CDE.
We defined documentation artifacts as instances in nursing documentation in which a finding unrelated to the patient’s acute health status, such as a scar, was newly documented as abnormal and led to worsening pRI. Any cases in which the clinical relevance was unclear underwent review by additional members of the team
To determine the temporal relationship among pRI, CDE, and clinician awareness or action, we then sought to systematically determine whether the preceding acuity alerts preceded documented evidence of clinicians recognizing deterioration or escalation of care. We made the a priori decision that acuity alerts that occurred more than 24 hours prior to a deterioration event had questionable clinical actionability. Therefore, we restricted this next analysis to CDEs with acuity alerts during the 24 hours prior to a CDE. We reviewed time-stamped progress notes written by clinicians in the 24 hours period prior to the time of the CDE and identified whether the notes reflected an adverse change in patient status or a clinical intervention. We then compared the times of these notes with the times of the alerts and CDEs. Given that documentation of change in clinical status often occurs after clinical intervention, we also reviewed new orders placed in the 24 hours prior to each CDE to determine escalation of care. We identified the following orders as reflective of escalation of care independent of specific disease process: administration of intravenous fluid bolus, blood product, steroid, or antibiotic, increased respiratory support, new imaging studies, and new laboratory studies. We then compared the time of each order with the time of the alert and CDE.
RESULTS
During the study period, 73 events met the CDE criteria and had a pRI alert during admission. Of the 73 events, 50 would have triggered at least one pRI alert in the 72-hour period leading up to the CDE (sensitivity 68%). Of the 50 events, 39 generated pRI alerts in the 24 hours leading up to the event, and 11 others generated pRI alerts between 24 and 72 hours prior to the event but did not generate any alerts during the 24 hours leading up to the event (Figure).
Patient Characteristics
The 50 CDEs labeled as true positives occurred in 46 unique patients. Table 1 displays the event characteristics.
Acuity Alerts
A total of 79 pRI alerts preceded the 50 CDEs. Of these acuity alerts, 44 (56%) were medium acuity alerts, 17 (22%) were high acuity alerts, and 18 (23%) were very high acuity alerts. Of the 50 CDEs that would have triggered pRI alerts, 33 (66%) would have triggered a single acuity alert and 17 (34%) would have triggered multiple acuity alerts.
Of the 50 CDEs, 39 (78%) had a preceding acuity alert within 24 hours prior to the CDE. In these cases, the alert preceded the CDE by a median of 3.1 hours (interquartile range of 0.7 to 10.3 hours).
We assessed the score components that caused each alert to trigger. All of the vital sign and laboratory components were assessed as clinically related to the CDE’s etiology. By contrast, about half of nursing assessment components were assessed as clinically related to the etiology of the CDE (Table 2). Abnormal cardiac, respiratory, and neurologic assessments were most frequently assessed as clinically relevant.
Escalation Orders
To determine whether the pRI alert would have preceded the earliest documented treatment efforts, we restricted evaluation to the 39 CDEs that had at least one alert in the 24-hour window prior to the CDE. When we reviewed escalation orders placed by clinicians, we found that in 26 cases (67%), the first clinician order reflecting escalation of care would have preceded the first pRI alert within the 24-hour period prior to the CDE. In 13 cases (33%), the first pRI alert would have preceded the first escalation order placed by the clinician. The first pRI alert and the first escalation order would have occurred within the same 1-hour period in 6 of these cases.
Provider Notes
Temporal Relationships
In Supplement 2, we present the proportion of CDEs in which the order or note preceded the pRI alert for each abnormal organ system.
The Figure shows the temporal relationships among escalation orders, clinician notes, and acuity alerts for the 39 CDEs with one or more alerts in the 24 hours leading up to the event. In 21 cases (54%), both an escalation order and a note preceded the first acuity alert. In 14 cases (36%), either an escalation order or a note preceded the first acuity alert. In four cases (10%), the alert preceded any documented evidence that clinicians had recognized deterioration or escalating care.
DISCUSSION
The main finding of this study is that 90% of CDE events that generated “true positive” pRI alerts had evidence suggesting that clinicians had already recognized deterioration and/or were already escalating care before most pRI alerts would have been triggered.
The impacts of early warning scores on patient safety outcomes are not well established. In a recent 21-hospital cluster randomized trial of the BedsidePEWS, a pediatric early warning score system, investigators found that implementing the system does not significantly decrease all-cause mortality in hospitalized children, although hospitals using the BedsidePEWS have low rates of significant CDEs.10 In other studies, early warning scores were often coimplemented with rapid response teams, and separating the incremental benefit of the scoring tool from the availability of a rapid response team is usually not possible.11
Therefore, the benefits of early warning scores are often inferred based on their test characteristics (eg, sensitivity and positive predictive value).12 Sensitivity, which is the proportion of patients who deteriorated and also triggered the early warning score within a reasonable time window preceding the event, is an important consideration when deciding whether an early warning score is worth implementing. A challenging follow-up question that goes beyond sensitivity is how often an early warning score adds new knowledge by identifying patients on a path toward deterioration who were not yet recognized. This study is the first to address that follow-up question. Our results revealed that the score appeared to precede evidence of clinician recognition of deterioration in 10% of CDEs. In some patients, the alert could have contributed to a detection of deterioration that was not previously evident. In the portion of CDEs in which the alert and escalation order or note occurred within the same one-hour window, the alert could have been used as confirmation of clinical suspicion. Notably, we did not evaluate the 16 cases in which a CDE preceded any pRI alert because we chose to focus on “true positive” cases in which pRI alerts preceded CDEs. These events could have had timely recognition by clinicians that we did not capture, so these results may provide an overestimation of CDEs in which the pRI preceded clinician recognition.
Prior work has described a range of mechanisms by which early warning scores can impact patient safety.13 The results of this study suggest limited incremental benefit for the pRI to alert physicians and nurses to new concerning changes at this hospital, although the benefits to low-resourced community hospitals that care for children may be great. The pRI score may also serve as evidence that empowers nurses to overcome barriers to further escalate care, even if the process of escalation has already begun. In addition to empowering nurses, the score may support trainees and clinicians with varying levels of pediatric expertise in the decision to escalate care. Evaluating these potential benefits would require prospective study.
We used the pRI alerts as they were already defined by PeraHealth for CHOP, and different alert thresholds may change score performance. Our study did not identify additional variables to improve score performance, but they can be investigated in future research.
This study had several limitations. First, this work is a single-center study with highly skilled pediatric providers, a mature rapid response system, and low rates of cardiopulmonary arrest outside ICUs. Therefore, the results that we obtained were not immediately generalizable. In a community environment with nurses and physicians who are less experienced in caring for ill children, an early warning score with high sensitivity may be beneficial in ensuring patient safety.
Second, by using escalation orders and notes from the patient chart, we did not capture all the undocumented ways in which clinicians demonstrate awareness of deterioration. For example, a resident may alert the attending on service or a team may informally request consultation with a specialist. We also gave equal weight to escalation orders and clinician notes as evidence of recognition of deterioration. It could be that either orders or notes more closely correlated with clinician awareness.
Finally, the data were from 2013. Although the score components have not changed, efforts to standardize nursing assessments may have altered the performance of the score in the intervening years.
CONCLUSIONS
In most patients who had a CDE at a large freestanding children’s hospital, escalation orders or documented changes in patient status would have occurred before a pRI alert. However, in a minority of patients, the alert could have contributed to the detection of deterioration that was not previously evident.
Disclosures
The authors have nothing to disclose
Funding
The study was supported by funds from the Department of Biomedical and Health Informatics at Children’s Hospital of Philadelphia. PeraHealth, the company that sells the Rothman Index software, provided a service to the investigators but no funding. They applied their proprietary scoring algorithm to the data from Children’s Hospital of Philadelphia to generate alerts retrospectively. This service was provided free of charge in 2014 during the time period when Children’s Hospital of Philadelphia was considering purchasing and implementing PeraHealth software, which it subsequently did. We did not receive any funding for the study from PeraHealth. PeraHealth personnel did not influence the study design, the interpretation of data, the writing of the report, or the decision to submit the article for publication.
Patients at risk for clinical deterioration in the inpatient setting may not be identified efficiently or effectively by health care providers. Early warning systems that link clinical observations to rapid response mechanisms (such as medical emergency teams) have the potential to improve outcomes, but rigorous studies are lacking.1 The pediatric Rothman Index (pRI) is an automated early warning system sold by the company PeraHealth that is integrated with the electronic health record. The system incorporates vital signs, labs, and nursing assessments from existing electronic health record data to provide a single numeric score that generates alerts based on low absolute scores and acute decreases in score (low scores indicate high mortality risk).2 Automated alerts or rules based on the pRI score are meant to bring important changes in clinical status to the attention of clinicians.
Adverse outcomes (eg, unplanned intensive care unit [ICU] transfers and mortality) are associated with low pRI scores, and scores appear to decline prior to such events.2 However, the limitation of this and other studies evaluating the sensitivity of early warning systems3-6 is that the generated alerts are assigned “true positive” status if they precede clinical deterioration, regardless of whether or not they provide meaningful information to the clinicians caring for the patients. There are two potential critiques of this approach. First, the alert may have preceded a deterioration event but may not have been clinically relevant (eg, an alert triggered by a finding unrelated to the patient’s acute health status, such as a scar that was newly documented as an abnormal skin finding and as a result led to a worsening in the pRI). Second, even if the preceding alert demonstrated clinical relevance to a deterioration event, the clinicians at the bedside may have been aware of the patient’s deterioration for hours and have already escalated care. In this situation, the alert would simply confirm what the clinician already knew.
To better understand the relationship between early warning system acuity alerts and clinical practice, we examined a cohort of hospitalized patients who experienced a critical deterioration event (CDE)7 and who would have triggered a preceding pRI alert. We evaluated the clinical relationship of the alert to the CDE (ie, whether the alert reflected physiologic changes related to a CDE or was instead an artifact of documentation) and identified whether the alert would have preceded evidence that clinicians recognized deterioration or escalated care.
METHODS
Patients and Setting
This retrospective cross-sectional study was performed at Children’s Hospital of Philadelphia (CHOP), a freestanding children’s hospital with 546 beds. Eligible patients were hospitalized on nonintensive care, noncardiology, surgical wards between January 1, 2013, and December 31, 2013. The CHOP Institutional Review Board (IRB) approved the study with waivers of consent and assent. A HIPAA Business Associate Agreement and an IRB Reliance Agreement were in place with PeraHealth to permit data transfer.
Definition of Critical Deterioration Events
Critical deterioration events (CDEs) were defined according to an existing, validated measure7 as unplanned transfers to the ICU with continuous or bilevel positive airway pressure, tracheal intubation, and/or vasopressor infusion in the 12 hours after transfer. At CHOP, all unplanned ICU transfers are routed through the hospital’s rapid response or code blue teams, so these patients were identified using an existing database managed by the CHOP Resuscitation Committee. In the database, the elements of CDEs are entered as part of ongoing quality improvement activities. The time of CDE was defined as the time of the rapid response call precipitating unplanned transfer to the ICU.
The Pediatric Rothman Index
The pRI is an automated acuity score that has been validated in hospitalized pediatric patients.2 The pRI is calculated using existing variables from the electronic health record, including manually entered vital signs, laboratory values, cardiac rhythm, and nursing assessments of organ systems. The weights assigned to continuous variables are a function of deviation from the norm.2,8 (See Supplement 1 for a complete list of variables.)
The pRI is integrated with the electronic health record and automatically generates a score each time a new data observation becomes available. Changes in score over time and low absolute scores generate a graduated series of alerts ranging from medium to very high acuity. This analysis used PeraHealth’s standard pRI alerts. Medium acuity alerts occurred when the pRI score decreased by ≥30% in 24 hours. A high acuity alert occurred when the pRI score decreased by ≥40% in 6 hours. A very high acuity alert occurred when the pRI absolute score was ≤ 30.
Development of the Source Dataset
In 2014, CHOP shared one year of clinical data with PeraHealth as part of the process of deciding whether or not to implement the pRI. The pRI algorithm retrospectively generated scores and acuity alerts for all CHOP patients who experienced CDEs between January 1, 2013, and December 31, 2013. The pRI algorithm was not active in the hospital environment during this time period; the scores and acuity alerts were not visible to clinicians. This dataset was provided to the investigators at CHOP to conduct this project.
Data Collection
Pediatric intensive care nurses trained in clinical research data abstraction from the CHOP Critical Care Center for Evidence and Outcomes performed the chart review for this study. Chart abstraction comparisons were completed on the first 15 charts to ensure interrater reliability, and additional quality assurance checks were performed on intermittent charts to ensure consistency and definition adherence. We managed all data using Research Electronic Data Capture.9
To study the value of alerts labeled as “true positives,” we restricted the dataset to CDEs in which acuity alert(s) within the prior 72 hours would have been triggered if the pRI had been in clinical use at the time.
To identify the clinical relationship between pRI and CDE, we reviewed each chart with the goal of determining whether the preceding acuity alerts were clinically associated with the etiology of the CDE. We determined the etiology of the CDE by reviewing the cause(s) identified in the note written by rapid response or code blue team responders or by the admitting clinical team after transfer to the ICU. We then used a tool provided by PeraHealth to identify the specific score components that led to worsening pRI. If the score components that worsened were (a) consistent with a clinical change as opposed to a documentation artifact and (b) an organ system change that was plausibly related to the CDE etiology, we concluded that the alert was clinically related to the etiology of the CDE.
We defined documentation artifacts as instances in nursing documentation in which a finding unrelated to the patient’s acute health status, such as a scar, was newly documented as abnormal and led to worsening pRI. Any cases in which the clinical relevance was unclear underwent review by additional members of the team
To determine the temporal relationship among pRI, CDE, and clinician awareness or action, we then sought to systematically determine whether the preceding acuity alerts preceded documented evidence of clinicians recognizing deterioration or escalation of care. We made the a priori decision that acuity alerts that occurred more than 24 hours prior to a deterioration event had questionable clinical actionability. Therefore, we restricted this next analysis to CDEs with acuity alerts during the 24 hours prior to a CDE. We reviewed time-stamped progress notes written by clinicians in the 24 hours period prior to the time of the CDE and identified whether the notes reflected an adverse change in patient status or a clinical intervention. We then compared the times of these notes with the times of the alerts and CDEs. Given that documentation of change in clinical status often occurs after clinical intervention, we also reviewed new orders placed in the 24 hours prior to each CDE to determine escalation of care. We identified the following orders as reflective of escalation of care independent of specific disease process: administration of intravenous fluid bolus, blood product, steroid, or antibiotic, increased respiratory support, new imaging studies, and new laboratory studies. We then compared the time of each order with the time of the alert and CDE.
RESULTS
During the study period, 73 events met the CDE criteria and had a pRI alert during admission. Of the 73 events, 50 would have triggered at least one pRI alert in the 72-hour period leading up to the CDE (sensitivity 68%). Of the 50 events, 39 generated pRI alerts in the 24 hours leading up to the event, and 11 others generated pRI alerts between 24 and 72 hours prior to the event but did not generate any alerts during the 24 hours leading up to the event (Figure).
Patient Characteristics
The 50 CDEs labeled as true positives occurred in 46 unique patients. Table 1 displays the event characteristics.
Acuity Alerts
A total of 79 pRI alerts preceded the 50 CDEs. Of these acuity alerts, 44 (56%) were medium acuity alerts, 17 (22%) were high acuity alerts, and 18 (23%) were very high acuity alerts. Of the 50 CDEs that would have triggered pRI alerts, 33 (66%) would have triggered a single acuity alert and 17 (34%) would have triggered multiple acuity alerts.
Of the 50 CDEs, 39 (78%) had a preceding acuity alert within 24 hours prior to the CDE. In these cases, the alert preceded the CDE by a median of 3.1 hours (interquartile range of 0.7 to 10.3 hours).
We assessed the score components that caused each alert to trigger. All of the vital sign and laboratory components were assessed as clinically related to the CDE’s etiology. By contrast, about half of nursing assessment components were assessed as clinically related to the etiology of the CDE (Table 2). Abnormal cardiac, respiratory, and neurologic assessments were most frequently assessed as clinically relevant.
Escalation Orders
To determine whether the pRI alert would have preceded the earliest documented treatment efforts, we restricted evaluation to the 39 CDEs that had at least one alert in the 24-hour window prior to the CDE. When we reviewed escalation orders placed by clinicians, we found that in 26 cases (67%), the first clinician order reflecting escalation of care would have preceded the first pRI alert within the 24-hour period prior to the CDE. In 13 cases (33%), the first pRI alert would have preceded the first escalation order placed by the clinician. The first pRI alert and the first escalation order would have occurred within the same 1-hour period in 6 of these cases.
Provider Notes
Temporal Relationships
In Supplement 2, we present the proportion of CDEs in which the order or note preceded the pRI alert for each abnormal organ system.
The Figure shows the temporal relationships among escalation orders, clinician notes, and acuity alerts for the 39 CDEs with one or more alerts in the 24 hours leading up to the event. In 21 cases (54%), both an escalation order and a note preceded the first acuity alert. In 14 cases (36%), either an escalation order or a note preceded the first acuity alert. In four cases (10%), the alert preceded any documented evidence that clinicians had recognized deterioration or escalating care.
DISCUSSION
The main finding of this study is that 90% of CDE events that generated “true positive” pRI alerts had evidence suggesting that clinicians had already recognized deterioration and/or were already escalating care before most pRI alerts would have been triggered.
The impacts of early warning scores on patient safety outcomes are not well established. In a recent 21-hospital cluster randomized trial of the BedsidePEWS, a pediatric early warning score system, investigators found that implementing the system does not significantly decrease all-cause mortality in hospitalized children, although hospitals using the BedsidePEWS have low rates of significant CDEs.10 In other studies, early warning scores were often coimplemented with rapid response teams, and separating the incremental benefit of the scoring tool from the availability of a rapid response team is usually not possible.11
Therefore, the benefits of early warning scores are often inferred based on their test characteristics (eg, sensitivity and positive predictive value).12 Sensitivity, which is the proportion of patients who deteriorated and also triggered the early warning score within a reasonable time window preceding the event, is an important consideration when deciding whether an early warning score is worth implementing. A challenging follow-up question that goes beyond sensitivity is how often an early warning score adds new knowledge by identifying patients on a path toward deterioration who were not yet recognized. This study is the first to address that follow-up question. Our results revealed that the score appeared to precede evidence of clinician recognition of deterioration in 10% of CDEs. In some patients, the alert could have contributed to a detection of deterioration that was not previously evident. In the portion of CDEs in which the alert and escalation order or note occurred within the same one-hour window, the alert could have been used as confirmation of clinical suspicion. Notably, we did not evaluate the 16 cases in which a CDE preceded any pRI alert because we chose to focus on “true positive” cases in which pRI alerts preceded CDEs. These events could have had timely recognition by clinicians that we did not capture, so these results may provide an overestimation of CDEs in which the pRI preceded clinician recognition.
Prior work has described a range of mechanisms by which early warning scores can impact patient safety.13 The results of this study suggest limited incremental benefit for the pRI to alert physicians and nurses to new concerning changes at this hospital, although the benefits to low-resourced community hospitals that care for children may be great. The pRI score may also serve as evidence that empowers nurses to overcome barriers to further escalate care, even if the process of escalation has already begun. In addition to empowering nurses, the score may support trainees and clinicians with varying levels of pediatric expertise in the decision to escalate care. Evaluating these potential benefits would require prospective study.
We used the pRI alerts as they were already defined by PeraHealth for CHOP, and different alert thresholds may change score performance. Our study did not identify additional variables to improve score performance, but they can be investigated in future research.
This study had several limitations. First, this work is a single-center study with highly skilled pediatric providers, a mature rapid response system, and low rates of cardiopulmonary arrest outside ICUs. Therefore, the results that we obtained were not immediately generalizable. In a community environment with nurses and physicians who are less experienced in caring for ill children, an early warning score with high sensitivity may be beneficial in ensuring patient safety.
Second, by using escalation orders and notes from the patient chart, we did not capture all the undocumented ways in which clinicians demonstrate awareness of deterioration. For example, a resident may alert the attending on service or a team may informally request consultation with a specialist. We also gave equal weight to escalation orders and clinician notes as evidence of recognition of deterioration. It could be that either orders or notes more closely correlated with clinician awareness.
Finally, the data were from 2013. Although the score components have not changed, efforts to standardize nursing assessments may have altered the performance of the score in the intervening years.
CONCLUSIONS
In most patients who had a CDE at a large freestanding children’s hospital, escalation orders or documented changes in patient status would have occurred before a pRI alert. However, in a minority of patients, the alert could have contributed to the detection of deterioration that was not previously evident.
Disclosures
The authors have nothing to disclose
Funding
The study was supported by funds from the Department of Biomedical and Health Informatics at Children’s Hospital of Philadelphia. PeraHealth, the company that sells the Rothman Index software, provided a service to the investigators but no funding. They applied their proprietary scoring algorithm to the data from Children’s Hospital of Philadelphia to generate alerts retrospectively. This service was provided free of charge in 2014 during the time period when Children’s Hospital of Philadelphia was considering purchasing and implementing PeraHealth software, which it subsequently did. We did not receive any funding for the study from PeraHealth. PeraHealth personnel did not influence the study design, the interpretation of data, the writing of the report, or the decision to submit the article for publication.
1. Alam N, Hobbelink EL, van Tienhoven AJ, van de Ven PM, Jansma EP, Nanayakkara PWB. The impact of the use of the Early Warning Score (EWS) on patient outcomes: a systematic review. Resuscitation. 2014;85(5):587-594. doi: 10.1016/j.resuscitation.2014.01.013. PubMed
2. Rothman MJ, Tepas JJ, Nowalk AJ, et al. Development and validation of a continuously age-adjusted measure of patient condition for hospitalized children using the electronic medical record. J Biomed Inform. 2017;66 (Supplement C):180-193. doi: 10.1016/j.jbi.2016.12.013. PubMed
3. Akre M, Finkelstein M, Erickson M, Liu M, Vanderbilt L, Billman G. Sensitivity of the pediatric early warning score to identify patient deterioration. Pediatrics. 2010;125(4):e763-e769. doi: 10.1542/peds.2009-0338. PubMed
4. Seiger N, Maconochie I, Oostenbrink R, Moll HA. Validity of different pediatric early warning scores in the emergency department. Pediatrics. 2013;132(4):e841-e850. doi: 10.1542/peds.2012-3594. PubMed
5. Parshuram CS, Hutchison J, Middaugh K. Development and initial validation of the Bedside Paediatric Early Warning System score. Crit Care Lond Engl. 2009;13(4):R135. doi: 10.1186/cc7998. PubMed
6. Hollis RH, Graham LA, Lazenby JP, et al. A role for the early warning score in early identification of critical postoperative complications. Ann Surg. 2016;263(5):918-923. doi: 10.1097/SLA.0000000000001514. PubMed
7. Bonafide CP, Roberts KE, Priestley MA, et al. Development of a pragmatic measure for evaluating and optimizing rapid response systems. Pediatrics. 2012;129(4):e874-e881. doi: 10.1542/peds.2011-2784. PubMed
8. Rothman MJ, Rothman SI, Beals J. Development and validation of a continuous measure of patient condition using the electronic medical record. J Biomed Inform. 2013;46(5):837-848. doi: 10.1016/j.jbi.2013.06.011. PubMed
9. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research Electronic Data Capture (REDCap) - A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377-381. doi: 10.1016/j.jbi.2008.08.010. PubMed
10. Parshuram CS, Dryden-Palmer K, Farrell C, et al. Effect of a pediatric early warning system on all-cause mortality in hospitalized pediatric patients: the EPOCH randomized clinical trial. JAMA. 2018;319(10):1002-1012. doi: 10.1001/jama.2018.0948. PubMed
11. Bonafide CP, Localio AR, Roberts KE, Nadkarni VM, Weirich CM, Keren R. Impact of rapid response system implementation on critical deterioration events in children. JAMA Pediatr. 2014;168(1):25-33. doi: 10.1001/jamapediatrics.2013.3266. PubMed
12. Romero-Brufau S, Huddleston JM, Escobar GJ, Liebow M. Why the C-statistic is not informative to evaluate early warning scores and what metrics to use. Crit Care. 2015;19:285. doi: 10.1186/s13054-015-0999-1. PubMed
13. Bonafide CP, Roberts KE, Weirich CM, et al. Beyond statistical prediction: qualitative evaluation of the mechanisms by which pediatric early warning scores impact patient safety. J Hosp Med. 2013;8(5):248-253. doi: 10.1002/jhm.2026. PubMed
1. Alam N, Hobbelink EL, van Tienhoven AJ, van de Ven PM, Jansma EP, Nanayakkara PWB. The impact of the use of the Early Warning Score (EWS) on patient outcomes: a systematic review. Resuscitation. 2014;85(5):587-594. doi: 10.1016/j.resuscitation.2014.01.013. PubMed
2. Rothman MJ, Tepas JJ, Nowalk AJ, et al. Development and validation of a continuously age-adjusted measure of patient condition for hospitalized children using the electronic medical record. J Biomed Inform. 2017;66 (Supplement C):180-193. doi: 10.1016/j.jbi.2016.12.013. PubMed
3. Akre M, Finkelstein M, Erickson M, Liu M, Vanderbilt L, Billman G. Sensitivity of the pediatric early warning score to identify patient deterioration. Pediatrics. 2010;125(4):e763-e769. doi: 10.1542/peds.2009-0338. PubMed
4. Seiger N, Maconochie I, Oostenbrink R, Moll HA. Validity of different pediatric early warning scores in the emergency department. Pediatrics. 2013;132(4):e841-e850. doi: 10.1542/peds.2012-3594. PubMed
5. Parshuram CS, Hutchison J, Middaugh K. Development and initial validation of the Bedside Paediatric Early Warning System score. Crit Care Lond Engl. 2009;13(4):R135. doi: 10.1186/cc7998. PubMed
6. Hollis RH, Graham LA, Lazenby JP, et al. A role for the early warning score in early identification of critical postoperative complications. Ann Surg. 2016;263(5):918-923. doi: 10.1097/SLA.0000000000001514. PubMed
7. Bonafide CP, Roberts KE, Priestley MA, et al. Development of a pragmatic measure for evaluating and optimizing rapid response systems. Pediatrics. 2012;129(4):e874-e881. doi: 10.1542/peds.2011-2784. PubMed
8. Rothman MJ, Rothman SI, Beals J. Development and validation of a continuous measure of patient condition using the electronic medical record. J Biomed Inform. 2013;46(5):837-848. doi: 10.1016/j.jbi.2013.06.011. PubMed
9. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research Electronic Data Capture (REDCap) - A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377-381. doi: 10.1016/j.jbi.2008.08.010. PubMed
10. Parshuram CS, Dryden-Palmer K, Farrell C, et al. Effect of a pediatric early warning system on all-cause mortality in hospitalized pediatric patients: the EPOCH randomized clinical trial. JAMA. 2018;319(10):1002-1012. doi: 10.1001/jama.2018.0948. PubMed
11. Bonafide CP, Localio AR, Roberts KE, Nadkarni VM, Weirich CM, Keren R. Impact of rapid response system implementation on critical deterioration events in children. JAMA Pediatr. 2014;168(1):25-33. doi: 10.1001/jamapediatrics.2013.3266. PubMed
12. Romero-Brufau S, Huddleston JM, Escobar GJ, Liebow M. Why the C-statistic is not informative to evaluate early warning scores and what metrics to use. Crit Care. 2015;19:285. doi: 10.1186/s13054-015-0999-1. PubMed
13. Bonafide CP, Roberts KE, Weirich CM, et al. Beyond statistical prediction: qualitative evaluation of the mechanisms by which pediatric early warning scores impact patient safety. J Hosp Med. 2013;8(5):248-253. doi: 10.1002/jhm.2026. PubMed
© 2018 Society of Hospital Medicine.