User login
Early Warning Score Qualitative Study
Thousands of hospitals have recently implemented rapid response systems (RRSs), attempting to reduce mortality outside of intensive care units (ICUs).[1, 2] These systems have 2 clinical components, a response (efferent) arm and an identification (afferent) arm.[3] The response arm is usually composed of a medical emergency team (MET) that responds to calls for urgent assistance. The identification arm includes tools to help clinicians recognize patients who require assistance from the MET. In many hospitals, the identification arm includes an early warning score (EWS). In pediatric patients, EWSs assign point values to vital signs that fall outside of age‐based ranges, among other clinical observations. They then generate a total score intended to help clinicians identify patients exhibiting early signs of deterioration.[4, 5, 6, 7, 8, 9, 10, 11]
When experimentally applied to vital sign datasets, the test characteristics of pediatric EWSs in detecting clinical deterioration are highly variable across studies, with major tradeoffs between sensitivity, specificity, and predictive values that differ by outcome, score, and cut‐point (Table 1). This reflects the difficulty of identifying deteriorating patients using only objective measures. However, in real‐world settings, EWSs are used by clinicians in conjunction with their clinical judgment. We hypothesized that EWSs have benefits that extend beyond their ability to predict deterioration, and thus have value not demonstrated by test characteristics alone. In order to further explore this issue, we aimed to qualitatively evaluate mechanisms beyond their statistical ability to predict deterioration by which physicians and nurses use EWSs to support their decision making.
| Score and Citation | Outcome Measure | Score Cut‐point | Sens | Spec | PPV | NPV |
|---|---|---|---|---|---|---|
| ||||||
| Brighton Paediatric Early Warning Score[5] | RRT or code blue call | 4 | 86% | NR | NR | NR |
| Bristol Paediatric Early Warning Tool[6, 11] | Escalation to higher level of care | 1 | ER | ER | 63% | NR |
| Cardiff and Vale Paediatric Early Warning System[7] | Respiratory or cardiac arrest, HDU/ICU admission, or death | 1 | 89% | 64% | 2% | >99% |
| Bedside Paediatric Early Warning System score, original version[8] | Code blue call | 5 | 78% | 95% | 4% | NR |
| Bedside Paediatric Early Warning System score, simplified version[9] | Urgent ICU admission without a code blue call | 8 | 82% | 93% | NR | NR |
| Bedside Paediatric Early Warning System score, simplified version[10] | Urgent ICU admission or code blue call | 7 | 64% | 91% | 9% | NR |
METHODS
Overview
As 1 component of a larger study, we conducted semistructured interviews with nurses and physicians at The Children's Hospital of Philadelphia (CHOP) between May and October 2011. In separate subprojects using the same participants, the larger study also aimed to identify residual barriers to calling for urgent assistance and assess the role of families in the recognition of deterioration and MET activation.
Setting
The Children's Hospital of Philadelphia is an urban, tertiary‐care pediatric hospital with 504 beds. Surgical patients hospitalized outside of ICUs are cared for by surgeons and surgical nurses without pediatrician co‐management. Implementation of a RRS was prompted by serious safety events in which clinical deterioration was either not recognized or was recognized and not escalated. Prior to RRS implementation, a code blue team could be activated for patients in immediate need of resuscitation, or, for less‐urgent needs, a pediatric ICU fellow could be paged by physicians for informal consults.
A multidisciplinary team developed and pilot‐tested the RRS, then implemented it hospital‐wide in February 2010. Representing an aspect of a multipronged approach to improve safety culture, the RRS consisted of (1) an EWS based upon Parshuram's Bedside Paediatric Early Warning System,[8, 9, 10] calculated by hand on a paper form (see online supplementary content) at the same frequency as vital signs (usually every 4 hours), and (2) a 30‐minute response MET available for activation by any clinician for any concern, 24 hours per day, 7 days per week. Escalation guidelines included a prompt to activate the MET for a score that increased to the red zone (9). For concerns that could not wait 30 minutes, any hospital employee could activate the immediate‐response code blue team.
Utilization of the RRS at CHOP is high, with 23 calls to the MET per day and a combined MET/code‐blue team call rate of 27.8 per 1000 admissions.[12] Previously reported pediatric call rates range from 2.8 to 44.0, with a median of 9.6 per 1000 admissions across 6 studies.[13, 14, 15, 16, 17, 18, 19] Since implementation, there has been a statistically significant net reduction in critical deterioration events (unpublished data).
Participants
We recruited nurses and physicians who had recently cared for children age 18 years on general medical or surgical wards with false‐negative or false‐positive EWSs (instances when the score failed to predict deterioration). Recruitment ceased when we reached thematic data saturation (a qualitative research term for the point at which no new themes emerge with additional interviews).[20]
Data Collection
Through a detailed review of the relevant literature and consultation with experts, we developed a semistructured interview guide (see online supplementary content) to elicit nurses' and physicians' viewpoints regarding the mechanisms by which they use EWSs to support their decision making.
Experienced qualitative research scientists (F.K.B. and J.H.H.) trained 2 study interviewers (B.P. and K.M.T.). In order to minimize social‐desirability bias, the interviewers were not clinicians and were not involved in RRS operations. Each interview was recorded, professionally transcribed, and imported into NVivo 8.0 software for analysis (QSR International, Melbourne, Australia).
Data Analysis
We coded the interviews inductively, without using a predetermined set of themes. This approach is known as grounded theory methodology.[21] Two team members coded each interview independently. They then reviewed their coding together and discussed discrepancies until reaching consensus. In weekly meetings while the interviews were ongoing, we compared newly collected data with themes that had previously emerged in order to guide further thematic development and refinement (the constant comparative method).[22] After all of the interviews were completed and consensus had been reached for each individual interview, the study team convened a series of additional meetings to further refine and finalize the themes.
Human Subjects
The CHOP Institutional Review Board approved this study. All participants provided written informed consent.
RESULTS
Participants
We recruited 27 nurses and 30 physicians before reaching thematic data saturation. Because surgical patients are underrepresented relative to medical patients among the population with false‐positive and false‐negative scores in our hospital, this included 3 randomly selected surgical nurses and 7 randomly selected surgical physicians recruited to ensure thematic data saturation for surgical settings. Characteristics of the participants are displayed in Table 2.
| Physicians (n=30) | Nurses (n=27) | |||
|---|---|---|---|---|
| ||||
| N | % | N | % | |
| Race | ||||
| Asian | 2 | 6.7 | 1 | 3.7 |
| Black | 0 | 0.0 | 2 | 7.4 |
| White | 26 | 86.7 | 22 | 81.5 |
| Prefer not to say | 1 | 3.3 | 1 | 3.7 |
| >1 race | 1 | 3.3 | 1 | 3.7 |
| Ethnicity | ||||
| Hispanic/Latino | 2 | 6.7 | 1 | 3.7 |
| Not Hispanic/Latino | 23 | 76.7 | 25 | 92.6 |
| Prefer not to say | 5 | 16.7 | 1 | 3.7 |
| Sex | ||||
| F | 16 | 53.3 | 25 | 92.6 |
| M | 14 | 46.7 | 2 | 7.4 |
| Practice setting | ||||
| Medical | 21 | 70.0 | 22 | 81.5 |
| Surgical | 9 | 30.0 | 5 | 18.5 |
| Among physicians only, experience level | ||||
| Intern | 7 | 23.3 | ||
| Senior resident | 7 | 23.3 | ||
| Attending physician | 16 | 53.3 | ||
| Among attending physicians only, no. of years practicing | ||||
| <5 | 8 | 50.0 | ||
| 5<10 | 3 | 18.8 | ||
| 10 | 5 | 31.3 | ||
| Among nurses only, no. of years practicing | ||||
| <1 | 5 | 18.5 | ||
| 1<2 | 5 | 18.5 | ||
| 2<5 | 9 | 33.3 | ||
| 5<10 | 4 | 14.8 | ||
| 10<20 | 1 | 3.7 | ||
| 20 | 3 | 11.1 | ||
| Recruitment method | ||||
| Cared for patient with false‐positive score | 10 | 33.3 | 14 | 51.9 |
| Cared for patient with false‐negative score | 13 | 43.3 | 10 | 37.0 |
| Randomly selected to ensure data saturation for surgical settings | 7 | 23.3 | 3 | 11.1 |
Thematic Analysis
We provide the final themes, associated subthemes, and representative quotations below, with additional supporting quotations in Table 3. Because CHOP's MET is named the Critical Assessment Team, the term CAT appears in some quotations.
|
| Theme 1: The EWS facilitates patient safety by alerting nurses and physicians to concerning vital sign changes and prompting them to think critically about the possibility of deterioration. |
| I think [the EWS] helps us to be focused and gives us definite criteria to look for if there is an issue or change. It hopefully gives us a head start if there is going to be a change. They have a way of tracking it with the different color‐coding system they use Like, Oh geez, the heart rate is a little bit higher, that changes the color from yellow to orange, then I have to let the charge nurse know because that is a change from where they were earlier it kind of organizes it, I feel like, from where it was before. (medical nurse with 23 years of experience) |
| I think for myself, as a new clinician, one of our main goals is to help judge sick versus not sick. So to have a concrete system for thinking about that is helpful. (medical intern) |
| I think [the EWS] can help put things together for us. When you are really busy, you don't always get to focus on a lot of details. It is like another red flag to say you might have not realized that the child's heart rates went up further, but now here's some evidence that they did. (medical senior resident) |
| I think that the ability to use the EWS to watch the progression of a patient over time is really helpful. I've had a few patients that have gotten sicker from a respiratory standpoint. We can have multiple on the floor at the same time, and what's nice is that sometimes nurses have been able to come to me and we can really see through the score that we are at the point where a higher level of care is needed, whereas, in the old system, without that, we would have had to essentially wait for true clinical decompensation before the ICU would have been involved. I think that does help to deliver better care. (medical senior resident) |
| Theme 2: The EWS provides less‐experienced nurses with helpful age‐based reference ranges for vital signs that they use when caring for hospitalized children. |
| Sometimes you just write down the vitals and maybe you are not really thinking, and then when you go to do the EWS you looked at the score and it's really off in their age range. It kind of gives you 1 more step to recognize that there's a problem. (medical nurse with <1 year of experience) |
| I see the role [of the EWS] more broadly as a guide of where your patient should fall with their vital signs according to their age. I think that has been the biggest help for me, to be able to visualize, I have a 3‐year‐old; this is where they should be for their respiratory rate or heart rate. I think it has been good to be able to see that they are falling within the range appropriate for their age. (surgical nurse with 9 years of experience) |
| Theme 3: The EWS provides concrete evidence of clinical changes in the form of a score. This empowers nurses to overcome escalation barriers and communicate their concerns, helping them take action to rescue their deteriorating patients. |
| The times when I think the EWS helped me out the most are when there is a little bit of disagreement maybe the doctors and the nurses don't see eye‐to‐eye on how the patient is doing and so a higher score can sometimes be a way to say, I know there is nothing specifically going on, but if you take a look at the EWSs they are turning up very significantly. That might be enough to at least get a second opinion on that patient or to start some kind of change in their care. (medical nurse with <1 year of experience) |
| If we have the EWS to back us up, we can use that to say, Look, I don't feel comfortable with this patient, the EWS is 7 and I know you are saying they are okay, but I would feel more comfortable calling. Having that protocol in place I feel like it really gives us a voice it kind of gives us the, not that they don't trust us, but if they say, Oh, I think the child is fine but if I tell them Look, their EWS is an 8 or a 9, they are like, Oh, okay. It is not just you freaking out. There is an issue. (medical nurse with 3 years of experience) |
| I think that since it has been instituted nursing is coming to residents more than they did beforehand Can you reassess this patient? Do you think that we should call CAT? I think that it encourages residents to reevaluate patients at times when things are changing, and quicker than it did without the system in place. (medical senior resident) |
| I view [the EWS] as a tool, like if I have someone managing my patients when I am on service this would be a good tool because it mandates the nurses to notify and it also mandates the residents to understand what's going on. I think that was done on purpose. (medical attending physician in practice for 8 years) |
| Theme 4: In some patients, the EWS may not help with decision‐making. These include patients who are very stable and have a low likelihood of deterioration, and patients with abnormal physiology at baseline who consistently have very high EWSs. |
| The patient I took care of in this situation was a really sick kid to begin with, and it wasn't so much they were concerned about his EWS because, unless there was a really serious event, he would probably be staying on our floor anyway in some cases we just have some really sick kids whose scores may constantly be high all the time, so it wouldn't be helpful for the doctors or us to really bring it up. (medical nurse with 1 year of experience) |
Of note, after interviewing 9 surgeons, we found that they were not very familiar with the EWS and had little to say either positively or negatively about the system. For example, when asked what they thought about the EWS, a surgical intern said, I have no idea. I don't have enough experience with it. This is probably the first time that I ever had anybody telling me that the system is in place. Therefore, surgeons did not contribute meaningfully to the themes below.
Theme 1: The EWS facilitates patient safety by alerting nurses and physicians to concerning vital sign changes and prompting them to think critically about the possibility of deterioration
Nurses and physicians frequently discussed the direct role of the EWS in revealing changes consistent with early signs of deterioration. A medical nurse with <1 year of experience said, The higher the number gets, the more it sets off a red flag to you to kind of keep an eye on certain things. They are just as important as taking a set of vitals. When asked if the EWS had ever helped to identify deterioration, a medical attending physician in practice for 5 years said, I think sometimes we will blow off, so to speak, certain things, but when you look at the numbers and you see a big [EWS] change versus if you were [just] looking at individual vital signs, then yeah, I think it has made a difference.
Nurses and physicians also discussed the role of the EWSs in prompting them to closely examine individual vital signs and think critically about whether or not a patient is exhibiting early signs of deterioration. A surgical nurse with <1 year of experience said, Sometimes I feel like if you want things to be okay you can kind of write them off, but when you have to write [the EWS] down it kind of jogs you to think, maybe something is going on or maybe someone else needs to know about this. A medical senior resident commented, I think it has alerted me earlier to changes in vital signs that I might not necessarily have known. I think there are nurses that use it and they see that there is an elevation and they call you about it. Then it makes me go back and look through and see what their vital signs are and if it happens in timewe only go through and look at everyone's vital signs about twice a dayit can be very helpful.
Theme 2: The EWS provides less‐experienced nurses with helpful age‐based reference ranges for vital signs that they use when caring for hospitalized children
Although this theme did not appear among physicians, nurses frequently noted that they referred to the scoring sheet as a reference for vital signs appropriate for hospitalized children. A surgical nurse with <1 year of experience said, In nursing school, I mostly dealt with adults. So, to figure out the different ranges for normal vital signs, it helps to have it listed on paper so I can see, 'Oh, I didn't realize that this 10‐year‐old's heart rate is higher than it should be.' A medical nurse with 14 years of experience cited the benefits for less‐experienced nurses, noting, [The EWS helps] newer nurses who don't know the ranges. Where it's Oh, my kid's blood pressure is 81 [mm Hg] over something, then they can look at their age and say, Oh, that is completely normal for a 2‐month‐old. But [before the EWS] there was nowhere to look to see the ranges. Unless you were [Pediatric Advanced Life Support] certified where you would know that stuff, there was a lot of anxiety related to vital signs.
Theme 3: The EWS provides concrete evidence of clinical changes in the form of a score. This empowers nurses to overcome escalation barriers and communicate their concerns, helping them take action to rescue their deteriorating patients
Nurses and physicians often described the role of the EWS as a source of objective evidence that a patient was exhibiting a concerning change. They shared the ways in which the EWS was used to convey concerns, noting most commonly that this was used as a communication tool by nurses to raise their concerns with physicians. A medical nurse with 23 years of experience said, [With the EWS] you feel like you have concrete evidence. It's not just a feeling [that] they are not looking as well as they were it feels scientific. Building upon this concept, a medical attending physician in practice for 2 years said, The EWS is a number that certainly gives people a sense of Here's the data behind why I am really coming to you and insisting on this. It is not calling and saying, I just have a bad feeling, it is, I have a bad feeling and his EWS has gone to a 9.
Theme 4: In some patients, the EWS may not help with decision making. These include patients who are very stable and have a low likelihood of deterioration, patients with abnormal physiology at baseline who consistently have very high EWSs, and patients experiencing neurologic deterioration
Nurses and physicians described some patient scenarios in which the EWS may not help with decision making. Discussing postoperative patients, a surgical nurse with 1 year of experience said, I love doing [the EWS] for some patients. I think it makes perfect sense. Then there are some patients [for whom] I am doing it just to do it because they are only here for 24 hours. They are completely stable. They never had 1 vital sign that was even a little bit off. It's kind of like we are just filling it out to fill it out. Commenting on patients at the other end of the spectrum, a medical attending physician in practice for 2 years said, [The EWS] can be a useful composite tool, but for specialty patients with abnormal baselines, I think it is much more a question of making sure you pay attention to the specific changes, whether it is the EWS or heart rate or vital signs or pain score or any of those things. A final area in which nurses and physicians identified weaknesses in the EWS surrounded neurologic deterioration. Specifically, nurses and physicians described experiences when the EWS increased minimally or not at all in patients with sudden seizures or concerning mental status changes that warranted escalation of care.
DISCUSSION
This study is the first to analyze viewpoints on the mechanisms by which EWSs impact decision making among physicians and nurses who had recently experienced score failures. Our study, performed in a children's hospital, builds upon the findings of related studies performed in hospitals that care primarily for adults.[23, 24, 25, 26, 27, 28] Andrews and Waterman found that nurses consider the utility of EWSs to extend beyond detecting deterioration by providing quantifiable evidence, packaged in the form of a score that improves communication between nurses and physicians.[23] Mackintosh and colleagues found that a RRS that included an EWS helped to formalize the way nurses and physicians understand deterioration, enable them to overcome hierarchical boundaries through structured discussions, and empower them to call for help.[24] In a quasi‐experimental study, McDonnell and colleagues found that an EWS improved self‐assessed knowledge, skills, and confidence of nursing staff to detect and manage deteriorating patients.[25] In addition, we describe novel findings, including the use of EWS parameters as reference ranges independent of the score, and specific situations when the EWS fails to support decision making. The weaknesses we identified could be used to drive EWS optimization for low‐risk patients who are stable as well as higher‐risk patients with abnormal baseline physiology and those at risk of neurologic deterioration.
This study has several limitations. Although the interviewers were not involved in RRS operations, it is possible that social desirability bias influenced responses. Next, we identified a knowledge gap among surgeons, and they contributed minimally to our findings. This is most likely because (1) surgical patients deteriorate on the wards less often than medical patients in our hospital, so surgeons are rarely presented with EWSs; (2) surgeons spend less time on the wards compared with medical physicians; and (3) surgical residents rotate in short blocks interspersed with rotations at other hospitals and may be less engaged in hospital safety initiatives.
CONCLUSIONS
Although EWSs perform only marginally well as statistical tools to predict clinical deterioration, nurses and physicians who recently experienced score failures described substantial benefits in using them to help identify deteriorating patients and transcend barriers to escalation of care by serving as objective communication tools. Combining an EWS with a clinician's judgment may result in a system better equipped to respond to deterioration than previous EWS studies focused on their test characteristics alone suggest. Future research should seek to compare and prospectively evaluate the clinical effectiveness of EWSs in real‐world settings.
Acknowledgments
Disclosures: This project was funded by the Pennsylvania Health Research Formula Fund Award (awarded to Keren and Bonafide) and the CHOP Nursing Research and Evidence‐Based Practice Award (awarded to Roberts). The funders did not influence the study design; the collection, analysis, or interpretation of data; the writing of the report; or the decision to submit the article for publication. The authors have no other conflicts to report.
- Institute for Healthcare Improvement. Overview of the Institute for Healthcare Improvement Five Million Lives Campaign. Available at: http://www.ihi.org/offerings/Initiatives/PastStrategicInitiatives/5MillionLivesCampaign/Pages/default.aspx. Accessed June 21, 2012.
- UK National Institute for Health and Clinical Excellence (NICE). Acutely Ill Patients in Hospital: Recognition of and Response to Acute Illness in Adults in Hospital. Available at: http://publications.nice.org.uk/acutely‐ill‐patients‐in‐hospital‐cg50. Published July 2007. Accessed June 21, 2012.
- , , , et al. Findings of the first consensus conference on medical emergency teams. Crit Care Med. 2006; 34(9):2463–2478.
- . Detecting and managing deterioration in children. Paediatr Nurs. 2005;17(1):32–35.
- , , , , , . Sensitivity of the Pediatric Early Warning Score to identify patient deterioration. Pediatrics. 2010;125(4):e763–e769.
- , , . Promoting care for acutely ill children—development and evaluation of a paediatric early warning tool. Intensive Crit Care Nurs. 2006;22(2):73–81.
- , , , . Prospective cohort study to test the predictability of the Cardiff and Vale paediatric early warning system. Arch Dis Child. 2009;94(8):602–606.
- , , . The Pediatric Early Warning System Score: a severity of illness score to predict urgent medical need in hospitalized children. J Crit Care. 2006;21(3):271–278.
- , , . Development and initial validation of the Bedside Paediatric Early Warning System score. Crit Care. 2009;13(4):R135.
- , , , et al. Multi‐centre validation of the Bedside Paediatric Early Warning System Score: a severity of illness score to detect evolving critical illness in hospitalized children. Crit Care. 2011;15(4):R184.
- , . Evaluation of a paediatric early warning tool—claims unsubstantiated. Intensive Crit Care Nurs. 2006;22(6):315–316.
- , , , et al. Development of a pragmatic measure for evaluating and optimizing rapid response systems. Pediatrics. 2012;129(4):e874–e881.
- , , , et al. Implementation of a multicenter rapid response system in pediatric academic hospitals is effective. Pediatrics. 2011;128(1):72–78.
- , , , , . Rapid response teams: a systematic review and meta‐analysis. Arch Intern Med. 2010;170(1):18–26.
- , , , et al. Implementation of a medical emergency team in a large pediatric teaching hospital prevents respiratory and cardiopulmonary arrests outside the intensive care unit. Pediatr Crit Care Med. 2007;8(3):236–246.
- , , , et al. Transition from a traditional code team to a medical emergency team and categorization of cardiopulmonary arrests in a children's center. Arch Pediatr Adolesc Med. 2008;162(2):117–122.
- , , , et al. Effect of a rapid response team on hospital‐wide mortality and code rates outside the ICU in a children's hospital. JAMA. 2007;298(19):2267–2274.
- , . Reduction of hospital mortality and of preventable cardiac arrest and death on introduction of a pediatric medical emergency team. Pediatr Crit Care Med. 2009;10(3):306–312.
- , , , et al. Implementation and impact of a rapid response team in a children's hospital. Jt Comm J Qual Patient Saf. 2007;33(7):418–425.
- , , . How many interviews are enough? An experiment with data saturation and variability. Field Methods. 2006;18(1):59–82.
- . Different approaches in grounded theory. In: Bryant A, Charmaz K, eds. The Sage Handbook of Grounded Theory. Los Angeles, CA: Sage; 2007:191–213.
- , . The Discovery of Grounded Theory: Strategies for Qualitative Research. New York, NY: Aldine De Gruyter; 1967.
- , . Packaging: a grounded theory of how to report physiological deterioration effectively. J Adv Nurs. 2005;52(5):473–481.
- , , . Understanding how rapid response systems may improve safety for the acutely ill patient: learning from the frontline. BMJ Qual Saf. 2012;21(2):135–144.
- , , , , , . A before and after study assessing the impact of a new model for recognizing and responding to early signs of deterioration in an acute hospital. J Adv Nurs. 2013;69(1):41–52.
- , . Overcoming gendered and professional hierarchies in order to facilitate escalation of care in emergency situations: the role of standardised communication protocols. Soc Sci Med. 2010;71(9):1683–1686.
- , , , , . Defining impact of a rapid response team: qualitative study with nurses, physicians and hospital administrators. BMJ Qual Saf. 2012;21(5):391–398.
- , , , , . Leading successful rapid response teams: a multisite implementation evaluation. J Nurs Adm. 2009;39(4):176–181.
Thousands of hospitals have recently implemented rapid response systems (RRSs), attempting to reduce mortality outside of intensive care units (ICUs).[1, 2] These systems have 2 clinical components, a response (efferent) arm and an identification (afferent) arm.[3] The response arm is usually composed of a medical emergency team (MET) that responds to calls for urgent assistance. The identification arm includes tools to help clinicians recognize patients who require assistance from the MET. In many hospitals, the identification arm includes an early warning score (EWS). In pediatric patients, EWSs assign point values to vital signs that fall outside of age‐based ranges, among other clinical observations. They then generate a total score intended to help clinicians identify patients exhibiting early signs of deterioration.[4, 5, 6, 7, 8, 9, 10, 11]
When experimentally applied to vital sign datasets, the test characteristics of pediatric EWSs in detecting clinical deterioration are highly variable across studies, with major tradeoffs between sensitivity, specificity, and predictive values that differ by outcome, score, and cut‐point (Table 1). This reflects the difficulty of identifying deteriorating patients using only objective measures. However, in real‐world settings, EWSs are used by clinicians in conjunction with their clinical judgment. We hypothesized that EWSs have benefits that extend beyond their ability to predict deterioration, and thus have value not demonstrated by test characteristics alone. In order to further explore this issue, we aimed to qualitatively evaluate mechanisms beyond their statistical ability to predict deterioration by which physicians and nurses use EWSs to support their decision making.
| Score and Citation | Outcome Measure | Score Cut‐point | Sens | Spec | PPV | NPV |
|---|---|---|---|---|---|---|
| ||||||
| Brighton Paediatric Early Warning Score[5] | RRT or code blue call | 4 | 86% | NR | NR | NR |
| Bristol Paediatric Early Warning Tool[6, 11] | Escalation to higher level of care | 1 | ER | ER | 63% | NR |
| Cardiff and Vale Paediatric Early Warning System[7] | Respiratory or cardiac arrest, HDU/ICU admission, or death | 1 | 89% | 64% | 2% | >99% |
| Bedside Paediatric Early Warning System score, original version[8] | Code blue call | 5 | 78% | 95% | 4% | NR |
| Bedside Paediatric Early Warning System score, simplified version[9] | Urgent ICU admission without a code blue call | 8 | 82% | 93% | NR | NR |
| Bedside Paediatric Early Warning System score, simplified version[10] | Urgent ICU admission or code blue call | 7 | 64% | 91% | 9% | NR |
METHODS
Overview
As 1 component of a larger study, we conducted semistructured interviews with nurses and physicians at The Children's Hospital of Philadelphia (CHOP) between May and October 2011. In separate subprojects using the same participants, the larger study also aimed to identify residual barriers to calling for urgent assistance and assess the role of families in the recognition of deterioration and MET activation.
Setting
The Children's Hospital of Philadelphia is an urban, tertiary‐care pediatric hospital with 504 beds. Surgical patients hospitalized outside of ICUs are cared for by surgeons and surgical nurses without pediatrician co‐management. Implementation of a RRS was prompted by serious safety events in which clinical deterioration was either not recognized or was recognized and not escalated. Prior to RRS implementation, a code blue team could be activated for patients in immediate need of resuscitation, or, for less‐urgent needs, a pediatric ICU fellow could be paged by physicians for informal consults.
A multidisciplinary team developed and pilot‐tested the RRS, then implemented it hospital‐wide in February 2010. Representing an aspect of a multipronged approach to improve safety culture, the RRS consisted of (1) an EWS based upon Parshuram's Bedside Paediatric Early Warning System,[8, 9, 10] calculated by hand on a paper form (see online supplementary content) at the same frequency as vital signs (usually every 4 hours), and (2) a 30‐minute response MET available for activation by any clinician for any concern, 24 hours per day, 7 days per week. Escalation guidelines included a prompt to activate the MET for a score that increased to the red zone (9). For concerns that could not wait 30 minutes, any hospital employee could activate the immediate‐response code blue team.
Utilization of the RRS at CHOP is high, with 23 calls to the MET per day and a combined MET/code‐blue team call rate of 27.8 per 1000 admissions.[12] Previously reported pediatric call rates range from 2.8 to 44.0, with a median of 9.6 per 1000 admissions across 6 studies.[13, 14, 15, 16, 17, 18, 19] Since implementation, there has been a statistically significant net reduction in critical deterioration events (unpublished data).
Participants
We recruited nurses and physicians who had recently cared for children age 18 years on general medical or surgical wards with false‐negative or false‐positive EWSs (instances when the score failed to predict deterioration). Recruitment ceased when we reached thematic data saturation (a qualitative research term for the point at which no new themes emerge with additional interviews).[20]
Data Collection
Through a detailed review of the relevant literature and consultation with experts, we developed a semistructured interview guide (see online supplementary content) to elicit nurses' and physicians' viewpoints regarding the mechanisms by which they use EWSs to support their decision making.
Experienced qualitative research scientists (F.K.B. and J.H.H.) trained 2 study interviewers (B.P. and K.M.T.). In order to minimize social‐desirability bias, the interviewers were not clinicians and were not involved in RRS operations. Each interview was recorded, professionally transcribed, and imported into NVivo 8.0 software for analysis (QSR International, Melbourne, Australia).
Data Analysis
We coded the interviews inductively, without using a predetermined set of themes. This approach is known as grounded theory methodology.[21] Two team members coded each interview independently. They then reviewed their coding together and discussed discrepancies until reaching consensus. In weekly meetings while the interviews were ongoing, we compared newly collected data with themes that had previously emerged in order to guide further thematic development and refinement (the constant comparative method).[22] After all of the interviews were completed and consensus had been reached for each individual interview, the study team convened a series of additional meetings to further refine and finalize the themes.
Human Subjects
The CHOP Institutional Review Board approved this study. All participants provided written informed consent.
RESULTS
Participants
We recruited 27 nurses and 30 physicians before reaching thematic data saturation. Because surgical patients are underrepresented relative to medical patients among the population with false‐positive and false‐negative scores in our hospital, this included 3 randomly selected surgical nurses and 7 randomly selected surgical physicians recruited to ensure thematic data saturation for surgical settings. Characteristics of the participants are displayed in Table 2.
| Physicians (n=30) | Nurses (n=27) | |||
|---|---|---|---|---|
| ||||
| N | % | N | % | |
| Race | ||||
| Asian | 2 | 6.7 | 1 | 3.7 |
| Black | 0 | 0.0 | 2 | 7.4 |
| White | 26 | 86.7 | 22 | 81.5 |
| Prefer not to say | 1 | 3.3 | 1 | 3.7 |
| >1 race | 1 | 3.3 | 1 | 3.7 |
| Ethnicity | ||||
| Hispanic/Latino | 2 | 6.7 | 1 | 3.7 |
| Not Hispanic/Latino | 23 | 76.7 | 25 | 92.6 |
| Prefer not to say | 5 | 16.7 | 1 | 3.7 |
| Sex | ||||
| F | 16 | 53.3 | 25 | 92.6 |
| M | 14 | 46.7 | 2 | 7.4 |
| Practice setting | ||||
| Medical | 21 | 70.0 | 22 | 81.5 |
| Surgical | 9 | 30.0 | 5 | 18.5 |
| Among physicians only, experience level | ||||
| Intern | 7 | 23.3 | ||
| Senior resident | 7 | 23.3 | ||
| Attending physician | 16 | 53.3 | ||
| Among attending physicians only, no. of years practicing | ||||
| <5 | 8 | 50.0 | ||
| 5<10 | 3 | 18.8 | ||
| 10 | 5 | 31.3 | ||
| Among nurses only, no. of years practicing | ||||
| <1 | 5 | 18.5 | ||
| 1<2 | 5 | 18.5 | ||
| 2<5 | 9 | 33.3 | ||
| 5<10 | 4 | 14.8 | ||
| 10<20 | 1 | 3.7 | ||
| 20 | 3 | 11.1 | ||
| Recruitment method | ||||
| Cared for patient with false‐positive score | 10 | 33.3 | 14 | 51.9 |
| Cared for patient with false‐negative score | 13 | 43.3 | 10 | 37.0 |
| Randomly selected to ensure data saturation for surgical settings | 7 | 23.3 | 3 | 11.1 |
Thematic Analysis
We provide the final themes, associated subthemes, and representative quotations below, with additional supporting quotations in Table 3. Because CHOP's MET is named the Critical Assessment Team, the term CAT appears in some quotations.
|
| Theme 1: The EWS facilitates patient safety by alerting nurses and physicians to concerning vital sign changes and prompting them to think critically about the possibility of deterioration. |
| I think [the EWS] helps us to be focused and gives us definite criteria to look for if there is an issue or change. It hopefully gives us a head start if there is going to be a change. They have a way of tracking it with the different color‐coding system they use Like, Oh geez, the heart rate is a little bit higher, that changes the color from yellow to orange, then I have to let the charge nurse know because that is a change from where they were earlier it kind of organizes it, I feel like, from where it was before. (medical nurse with 23 years of experience) |
| I think for myself, as a new clinician, one of our main goals is to help judge sick versus not sick. So to have a concrete system for thinking about that is helpful. (medical intern) |
| I think [the EWS] can help put things together for us. When you are really busy, you don't always get to focus on a lot of details. It is like another red flag to say you might have not realized that the child's heart rates went up further, but now here's some evidence that they did. (medical senior resident) |
| I think that the ability to use the EWS to watch the progression of a patient over time is really helpful. I've had a few patients that have gotten sicker from a respiratory standpoint. We can have multiple on the floor at the same time, and what's nice is that sometimes nurses have been able to come to me and we can really see through the score that we are at the point where a higher level of care is needed, whereas, in the old system, without that, we would have had to essentially wait for true clinical decompensation before the ICU would have been involved. I think that does help to deliver better care. (medical senior resident) |
| Theme 2: The EWS provides less‐experienced nurses with helpful age‐based reference ranges for vital signs that they use when caring for hospitalized children. |
| Sometimes you just write down the vitals and maybe you are not really thinking, and then when you go to do the EWS you looked at the score and it's really off in their age range. It kind of gives you 1 more step to recognize that there's a problem. (medical nurse with <1 year of experience) |
| I see the role [of the EWS] more broadly as a guide of where your patient should fall with their vital signs according to their age. I think that has been the biggest help for me, to be able to visualize, I have a 3‐year‐old; this is where they should be for their respiratory rate or heart rate. I think it has been good to be able to see that they are falling within the range appropriate for their age. (surgical nurse with 9 years of experience) |
| Theme 3: The EWS provides concrete evidence of clinical changes in the form of a score. This empowers nurses to overcome escalation barriers and communicate their concerns, helping them take action to rescue their deteriorating patients. |
| The times when I think the EWS helped me out the most are when there is a little bit of disagreement maybe the doctors and the nurses don't see eye‐to‐eye on how the patient is doing and so a higher score can sometimes be a way to say, I know there is nothing specifically going on, but if you take a look at the EWSs they are turning up very significantly. That might be enough to at least get a second opinion on that patient or to start some kind of change in their care. (medical nurse with <1 year of experience) |
| If we have the EWS to back us up, we can use that to say, Look, I don't feel comfortable with this patient, the EWS is 7 and I know you are saying they are okay, but I would feel more comfortable calling. Having that protocol in place I feel like it really gives us a voice it kind of gives us the, not that they don't trust us, but if they say, Oh, I think the child is fine but if I tell them Look, their EWS is an 8 or a 9, they are like, Oh, okay. It is not just you freaking out. There is an issue. (medical nurse with 3 years of experience) |
| I think that since it has been instituted nursing is coming to residents more than they did beforehand Can you reassess this patient? Do you think that we should call CAT? I think that it encourages residents to reevaluate patients at times when things are changing, and quicker than it did without the system in place. (medical senior resident) |
| I view [the EWS] as a tool, like if I have someone managing my patients when I am on service this would be a good tool because it mandates the nurses to notify and it also mandates the residents to understand what's going on. I think that was done on purpose. (medical attending physician in practice for 8 years) |
| Theme 4: In some patients, the EWS may not help with decision‐making. These include patients who are very stable and have a low likelihood of deterioration, and patients with abnormal physiology at baseline who consistently have very high EWSs. |
| The patient I took care of in this situation was a really sick kid to begin with, and it wasn't so much they were concerned about his EWS because, unless there was a really serious event, he would probably be staying on our floor anyway in some cases we just have some really sick kids whose scores may constantly be high all the time, so it wouldn't be helpful for the doctors or us to really bring it up. (medical nurse with 1 year of experience) |
Of note, after interviewing 9 surgeons, we found that they were not very familiar with the EWS and had little to say either positively or negatively about the system. For example, when asked what they thought about the EWS, a surgical intern said, I have no idea. I don't have enough experience with it. This is probably the first time that I ever had anybody telling me that the system is in place. Therefore, surgeons did not contribute meaningfully to the themes below.
Theme 1: The EWS facilitates patient safety by alerting nurses and physicians to concerning vital sign changes and prompting them to think critically about the possibility of deterioration
Nurses and physicians frequently discussed the direct role of the EWS in revealing changes consistent with early signs of deterioration. A medical nurse with <1 year of experience said, The higher the number gets, the more it sets off a red flag to you to kind of keep an eye on certain things. They are just as important as taking a set of vitals. When asked if the EWS had ever helped to identify deterioration, a medical attending physician in practice for 5 years said, I think sometimes we will blow off, so to speak, certain things, but when you look at the numbers and you see a big [EWS] change versus if you were [just] looking at individual vital signs, then yeah, I think it has made a difference.
Nurses and physicians also discussed the role of the EWSs in prompting them to closely examine individual vital signs and think critically about whether or not a patient is exhibiting early signs of deterioration. A surgical nurse with <1 year of experience said, Sometimes I feel like if you want things to be okay you can kind of write them off, but when you have to write [the EWS] down it kind of jogs you to think, maybe something is going on or maybe someone else needs to know about this. A medical senior resident commented, I think it has alerted me earlier to changes in vital signs that I might not necessarily have known. I think there are nurses that use it and they see that there is an elevation and they call you about it. Then it makes me go back and look through and see what their vital signs are and if it happens in timewe only go through and look at everyone's vital signs about twice a dayit can be very helpful.
Theme 2: The EWS provides less‐experienced nurses with helpful age‐based reference ranges for vital signs that they use when caring for hospitalized children
Although this theme did not appear among physicians, nurses frequently noted that they referred to the scoring sheet as a reference for vital signs appropriate for hospitalized children. A surgical nurse with <1 year of experience said, In nursing school, I mostly dealt with adults. So, to figure out the different ranges for normal vital signs, it helps to have it listed on paper so I can see, 'Oh, I didn't realize that this 10‐year‐old's heart rate is higher than it should be.' A medical nurse with 14 years of experience cited the benefits for less‐experienced nurses, noting, [The EWS helps] newer nurses who don't know the ranges. Where it's Oh, my kid's blood pressure is 81 [mm Hg] over something, then they can look at their age and say, Oh, that is completely normal for a 2‐month‐old. But [before the EWS] there was nowhere to look to see the ranges. Unless you were [Pediatric Advanced Life Support] certified where you would know that stuff, there was a lot of anxiety related to vital signs.
Theme 3: The EWS provides concrete evidence of clinical changes in the form of a score. This empowers nurses to overcome escalation barriers and communicate their concerns, helping them take action to rescue their deteriorating patients
Nurses and physicians often described the role of the EWS as a source of objective evidence that a patient was exhibiting a concerning change. They shared the ways in which the EWS was used to convey concerns, noting most commonly that this was used as a communication tool by nurses to raise their concerns with physicians. A medical nurse with 23 years of experience said, [With the EWS] you feel like you have concrete evidence. It's not just a feeling [that] they are not looking as well as they were it feels scientific. Building upon this concept, a medical attending physician in practice for 2 years said, The EWS is a number that certainly gives people a sense of Here's the data behind why I am really coming to you and insisting on this. It is not calling and saying, I just have a bad feeling, it is, I have a bad feeling and his EWS has gone to a 9.
Theme 4: In some patients, the EWS may not help with decision making. These include patients who are very stable and have a low likelihood of deterioration, patients with abnormal physiology at baseline who consistently have very high EWSs, and patients experiencing neurologic deterioration
Nurses and physicians described some patient scenarios in which the EWS may not help with decision making. Discussing postoperative patients, a surgical nurse with 1 year of experience said, I love doing [the EWS] for some patients. I think it makes perfect sense. Then there are some patients [for whom] I am doing it just to do it because they are only here for 24 hours. They are completely stable. They never had 1 vital sign that was even a little bit off. It's kind of like we are just filling it out to fill it out. Commenting on patients at the other end of the spectrum, a medical attending physician in practice for 2 years said, [The EWS] can be a useful composite tool, but for specialty patients with abnormal baselines, I think it is much more a question of making sure you pay attention to the specific changes, whether it is the EWS or heart rate or vital signs or pain score or any of those things. A final area in which nurses and physicians identified weaknesses in the EWS surrounded neurologic deterioration. Specifically, nurses and physicians described experiences when the EWS increased minimally or not at all in patients with sudden seizures or concerning mental status changes that warranted escalation of care.
DISCUSSION
This study is the first to analyze viewpoints on the mechanisms by which EWSs impact decision making among physicians and nurses who had recently experienced score failures. Our study, performed in a children's hospital, builds upon the findings of related studies performed in hospitals that care primarily for adults.[23, 24, 25, 26, 27, 28] Andrews and Waterman found that nurses consider the utility of EWSs to extend beyond detecting deterioration by providing quantifiable evidence, packaged in the form of a score that improves communication between nurses and physicians.[23] Mackintosh and colleagues found that a RRS that included an EWS helped to formalize the way nurses and physicians understand deterioration, enable them to overcome hierarchical boundaries through structured discussions, and empower them to call for help.[24] In a quasi‐experimental study, McDonnell and colleagues found that an EWS improved self‐assessed knowledge, skills, and confidence of nursing staff to detect and manage deteriorating patients.[25] In addition, we describe novel findings, including the use of EWS parameters as reference ranges independent of the score, and specific situations when the EWS fails to support decision making. The weaknesses we identified could be used to drive EWS optimization for low‐risk patients who are stable as well as higher‐risk patients with abnormal baseline physiology and those at risk of neurologic deterioration.
This study has several limitations. Although the interviewers were not involved in RRS operations, it is possible that social desirability bias influenced responses. Next, we identified a knowledge gap among surgeons, and they contributed minimally to our findings. This is most likely because (1) surgical patients deteriorate on the wards less often than medical patients in our hospital, so surgeons are rarely presented with EWSs; (2) surgeons spend less time on the wards compared with medical physicians; and (3) surgical residents rotate in short blocks interspersed with rotations at other hospitals and may be less engaged in hospital safety initiatives.
CONCLUSIONS
Although EWSs perform only marginally well as statistical tools to predict clinical deterioration, nurses and physicians who recently experienced score failures described substantial benefits in using them to help identify deteriorating patients and transcend barriers to escalation of care by serving as objective communication tools. Combining an EWS with a clinician's judgment may result in a system better equipped to respond to deterioration than previous EWS studies focused on their test characteristics alone suggest. Future research should seek to compare and prospectively evaluate the clinical effectiveness of EWSs in real‐world settings.
Acknowledgments
Disclosures: This project was funded by the Pennsylvania Health Research Formula Fund Award (awarded to Keren and Bonafide) and the CHOP Nursing Research and Evidence‐Based Practice Award (awarded to Roberts). The funders did not influence the study design; the collection, analysis, or interpretation of data; the writing of the report; or the decision to submit the article for publication. The authors have no other conflicts to report.
Thousands of hospitals have recently implemented rapid response systems (RRSs), attempting to reduce mortality outside of intensive care units (ICUs).[1, 2] These systems have 2 clinical components, a response (efferent) arm and an identification (afferent) arm.[3] The response arm is usually composed of a medical emergency team (MET) that responds to calls for urgent assistance. The identification arm includes tools to help clinicians recognize patients who require assistance from the MET. In many hospitals, the identification arm includes an early warning score (EWS). In pediatric patients, EWSs assign point values to vital signs that fall outside of age‐based ranges, among other clinical observations. They then generate a total score intended to help clinicians identify patients exhibiting early signs of deterioration.[4, 5, 6, 7, 8, 9, 10, 11]
When experimentally applied to vital sign datasets, the test characteristics of pediatric EWSs in detecting clinical deterioration are highly variable across studies, with major tradeoffs between sensitivity, specificity, and predictive values that differ by outcome, score, and cut‐point (Table 1). This reflects the difficulty of identifying deteriorating patients using only objective measures. However, in real‐world settings, EWSs are used by clinicians in conjunction with their clinical judgment. We hypothesized that EWSs have benefits that extend beyond their ability to predict deterioration, and thus have value not demonstrated by test characteristics alone. In order to further explore this issue, we aimed to qualitatively evaluate mechanisms beyond their statistical ability to predict deterioration by which physicians and nurses use EWSs to support their decision making.
| Score and Citation | Outcome Measure | Score Cut‐point | Sens | Spec | PPV | NPV |
|---|---|---|---|---|---|---|
| ||||||
| Brighton Paediatric Early Warning Score[5] | RRT or code blue call | 4 | 86% | NR | NR | NR |
| Bristol Paediatric Early Warning Tool[6, 11] | Escalation to higher level of care | 1 | ER | ER | 63% | NR |
| Cardiff and Vale Paediatric Early Warning System[7] | Respiratory or cardiac arrest, HDU/ICU admission, or death | 1 | 89% | 64% | 2% | >99% |
| Bedside Paediatric Early Warning System score, original version[8] | Code blue call | 5 | 78% | 95% | 4% | NR |
| Bedside Paediatric Early Warning System score, simplified version[9] | Urgent ICU admission without a code blue call | 8 | 82% | 93% | NR | NR |
| Bedside Paediatric Early Warning System score, simplified version[10] | Urgent ICU admission or code blue call | 7 | 64% | 91% | 9% | NR |
METHODS
Overview
As 1 component of a larger study, we conducted semistructured interviews with nurses and physicians at The Children's Hospital of Philadelphia (CHOP) between May and October 2011. In separate subprojects using the same participants, the larger study also aimed to identify residual barriers to calling for urgent assistance and assess the role of families in the recognition of deterioration and MET activation.
Setting
The Children's Hospital of Philadelphia is an urban, tertiary‐care pediatric hospital with 504 beds. Surgical patients hospitalized outside of ICUs are cared for by surgeons and surgical nurses without pediatrician co‐management. Implementation of a RRS was prompted by serious safety events in which clinical deterioration was either not recognized or was recognized and not escalated. Prior to RRS implementation, a code blue team could be activated for patients in immediate need of resuscitation, or, for less‐urgent needs, a pediatric ICU fellow could be paged by physicians for informal consults.
A multidisciplinary team developed and pilot‐tested the RRS, then implemented it hospital‐wide in February 2010. Representing an aspect of a multipronged approach to improve safety culture, the RRS consisted of (1) an EWS based upon Parshuram's Bedside Paediatric Early Warning System,[8, 9, 10] calculated by hand on a paper form (see online supplementary content) at the same frequency as vital signs (usually every 4 hours), and (2) a 30‐minute response MET available for activation by any clinician for any concern, 24 hours per day, 7 days per week. Escalation guidelines included a prompt to activate the MET for a score that increased to the red zone (9). For concerns that could not wait 30 minutes, any hospital employee could activate the immediate‐response code blue team.
Utilization of the RRS at CHOP is high, with 23 calls to the MET per day and a combined MET/code‐blue team call rate of 27.8 per 1000 admissions.[12] Previously reported pediatric call rates range from 2.8 to 44.0, with a median of 9.6 per 1000 admissions across 6 studies.[13, 14, 15, 16, 17, 18, 19] Since implementation, there has been a statistically significant net reduction in critical deterioration events (unpublished data).
Participants
We recruited nurses and physicians who had recently cared for children age 18 years on general medical or surgical wards with false‐negative or false‐positive EWSs (instances when the score failed to predict deterioration). Recruitment ceased when we reached thematic data saturation (a qualitative research term for the point at which no new themes emerge with additional interviews).[20]
Data Collection
Through a detailed review of the relevant literature and consultation with experts, we developed a semistructured interview guide (see online supplementary content) to elicit nurses' and physicians' viewpoints regarding the mechanisms by which they use EWSs to support their decision making.
Experienced qualitative research scientists (F.K.B. and J.H.H.) trained 2 study interviewers (B.P. and K.M.T.). In order to minimize social‐desirability bias, the interviewers were not clinicians and were not involved in RRS operations. Each interview was recorded, professionally transcribed, and imported into NVivo 8.0 software for analysis (QSR International, Melbourne, Australia).
Data Analysis
We coded the interviews inductively, without using a predetermined set of themes. This approach is known as grounded theory methodology.[21] Two team members coded each interview independently. They then reviewed their coding together and discussed discrepancies until reaching consensus. In weekly meetings while the interviews were ongoing, we compared newly collected data with themes that had previously emerged in order to guide further thematic development and refinement (the constant comparative method).[22] After all of the interviews were completed and consensus had been reached for each individual interview, the study team convened a series of additional meetings to further refine and finalize the themes.
Human Subjects
The CHOP Institutional Review Board approved this study. All participants provided written informed consent.
RESULTS
Participants
We recruited 27 nurses and 30 physicians before reaching thematic data saturation. Because surgical patients are underrepresented relative to medical patients among the population with false‐positive and false‐negative scores in our hospital, this included 3 randomly selected surgical nurses and 7 randomly selected surgical physicians recruited to ensure thematic data saturation for surgical settings. Characteristics of the participants are displayed in Table 2.
| Physicians (n=30) | Nurses (n=27) | |||
|---|---|---|---|---|
| ||||
| N | % | N | % | |
| Race | ||||
| Asian | 2 | 6.7 | 1 | 3.7 |
| Black | 0 | 0.0 | 2 | 7.4 |
| White | 26 | 86.7 | 22 | 81.5 |
| Prefer not to say | 1 | 3.3 | 1 | 3.7 |
| >1 race | 1 | 3.3 | 1 | 3.7 |
| Ethnicity | ||||
| Hispanic/Latino | 2 | 6.7 | 1 | 3.7 |
| Not Hispanic/Latino | 23 | 76.7 | 25 | 92.6 |
| Prefer not to say | 5 | 16.7 | 1 | 3.7 |
| Sex | ||||
| F | 16 | 53.3 | 25 | 92.6 |
| M | 14 | 46.7 | 2 | 7.4 |
| Practice setting | ||||
| Medical | 21 | 70.0 | 22 | 81.5 |
| Surgical | 9 | 30.0 | 5 | 18.5 |
| Among physicians only, experience level | ||||
| Intern | 7 | 23.3 | ||
| Senior resident | 7 | 23.3 | ||
| Attending physician | 16 | 53.3 | ||
| Among attending physicians only, no. of years practicing | ||||
| <5 | 8 | 50.0 | ||
| 5<10 | 3 | 18.8 | ||
| 10 | 5 | 31.3 | ||
| Among nurses only, no. of years practicing | ||||
| <1 | 5 | 18.5 | ||
| 1<2 | 5 | 18.5 | ||
| 2<5 | 9 | 33.3 | ||
| 5<10 | 4 | 14.8 | ||
| 10<20 | 1 | 3.7 | ||
| 20 | 3 | 11.1 | ||
| Recruitment method | ||||
| Cared for patient with false‐positive score | 10 | 33.3 | 14 | 51.9 |
| Cared for patient with false‐negative score | 13 | 43.3 | 10 | 37.0 |
| Randomly selected to ensure data saturation for surgical settings | 7 | 23.3 | 3 | 11.1 |
Thematic Analysis
We provide the final themes, associated subthemes, and representative quotations below, with additional supporting quotations in Table 3. Because CHOP's MET is named the Critical Assessment Team, the term CAT appears in some quotations.
|
| Theme 1: The EWS facilitates patient safety by alerting nurses and physicians to concerning vital sign changes and prompting them to think critically about the possibility of deterioration. |
| I think [the EWS] helps us to be focused and gives us definite criteria to look for if there is an issue or change. It hopefully gives us a head start if there is going to be a change. They have a way of tracking it with the different color‐coding system they use Like, Oh geez, the heart rate is a little bit higher, that changes the color from yellow to orange, then I have to let the charge nurse know because that is a change from where they were earlier it kind of organizes it, I feel like, from where it was before. (medical nurse with 23 years of experience) |
| I think for myself, as a new clinician, one of our main goals is to help judge sick versus not sick. So to have a concrete system for thinking about that is helpful. (medical intern) |
| I think [the EWS] can help put things together for us. When you are really busy, you don't always get to focus on a lot of details. It is like another red flag to say you might have not realized that the child's heart rates went up further, but now here's some evidence that they did. (medical senior resident) |
| I think that the ability to use the EWS to watch the progression of a patient over time is really helpful. I've had a few patients that have gotten sicker from a respiratory standpoint. We can have multiple on the floor at the same time, and what's nice is that sometimes nurses have been able to come to me and we can really see through the score that we are at the point where a higher level of care is needed, whereas, in the old system, without that, we would have had to essentially wait for true clinical decompensation before the ICU would have been involved. I think that does help to deliver better care. (medical senior resident) |
| Theme 2: The EWS provides less‐experienced nurses with helpful age‐based reference ranges for vital signs that they use when caring for hospitalized children. |
| Sometimes you just write down the vitals and maybe you are not really thinking, and then when you go to do the EWS you looked at the score and it's really off in their age range. It kind of gives you 1 more step to recognize that there's a problem. (medical nurse with <1 year of experience) |
| I see the role [of the EWS] more broadly as a guide of where your patient should fall with their vital signs according to their age. I think that has been the biggest help for me, to be able to visualize, I have a 3‐year‐old; this is where they should be for their respiratory rate or heart rate. I think it has been good to be able to see that they are falling within the range appropriate for their age. (surgical nurse with 9 years of experience) |
| Theme 3: The EWS provides concrete evidence of clinical changes in the form of a score. This empowers nurses to overcome escalation barriers and communicate their concerns, helping them take action to rescue their deteriorating patients. |
| The times when I think the EWS helped me out the most are when there is a little bit of disagreement maybe the doctors and the nurses don't see eye‐to‐eye on how the patient is doing and so a higher score can sometimes be a way to say, I know there is nothing specifically going on, but if you take a look at the EWSs they are turning up very significantly. That might be enough to at least get a second opinion on that patient or to start some kind of change in their care. (medical nurse with <1 year of experience) |
| If we have the EWS to back us up, we can use that to say, Look, I don't feel comfortable with this patient, the EWS is 7 and I know you are saying they are okay, but I would feel more comfortable calling. Having that protocol in place I feel like it really gives us a voice it kind of gives us the, not that they don't trust us, but if they say, Oh, I think the child is fine but if I tell them Look, their EWS is an 8 or a 9, they are like, Oh, okay. It is not just you freaking out. There is an issue. (medical nurse with 3 years of experience) |
| I think that since it has been instituted nursing is coming to residents more than they did beforehand Can you reassess this patient? Do you think that we should call CAT? I think that it encourages residents to reevaluate patients at times when things are changing, and quicker than it did without the system in place. (medical senior resident) |
| I view [the EWS] as a tool, like if I have someone managing my patients when I am on service this would be a good tool because it mandates the nurses to notify and it also mandates the residents to understand what's going on. I think that was done on purpose. (medical attending physician in practice for 8 years) |
| Theme 4: In some patients, the EWS may not help with decision‐making. These include patients who are very stable and have a low likelihood of deterioration, and patients with abnormal physiology at baseline who consistently have very high EWSs. |
| The patient I took care of in this situation was a really sick kid to begin with, and it wasn't so much they were concerned about his EWS because, unless there was a really serious event, he would probably be staying on our floor anyway in some cases we just have some really sick kids whose scores may constantly be high all the time, so it wouldn't be helpful for the doctors or us to really bring it up. (medical nurse with 1 year of experience) |
Of note, after interviewing 9 surgeons, we found that they were not very familiar with the EWS and had little to say either positively or negatively about the system. For example, when asked what they thought about the EWS, a surgical intern said, I have no idea. I don't have enough experience with it. This is probably the first time that I ever had anybody telling me that the system is in place. Therefore, surgeons did not contribute meaningfully to the themes below.
Theme 1: The EWS facilitates patient safety by alerting nurses and physicians to concerning vital sign changes and prompting them to think critically about the possibility of deterioration
Nurses and physicians frequently discussed the direct role of the EWS in revealing changes consistent with early signs of deterioration. A medical nurse with <1 year of experience said, The higher the number gets, the more it sets off a red flag to you to kind of keep an eye on certain things. They are just as important as taking a set of vitals. When asked if the EWS had ever helped to identify deterioration, a medical attending physician in practice for 5 years said, I think sometimes we will blow off, so to speak, certain things, but when you look at the numbers and you see a big [EWS] change versus if you were [just] looking at individual vital signs, then yeah, I think it has made a difference.
Nurses and physicians also discussed the role of the EWSs in prompting them to closely examine individual vital signs and think critically about whether or not a patient is exhibiting early signs of deterioration. A surgical nurse with <1 year of experience said, Sometimes I feel like if you want things to be okay you can kind of write them off, but when you have to write [the EWS] down it kind of jogs you to think, maybe something is going on or maybe someone else needs to know about this. A medical senior resident commented, I think it has alerted me earlier to changes in vital signs that I might not necessarily have known. I think there are nurses that use it and they see that there is an elevation and they call you about it. Then it makes me go back and look through and see what their vital signs are and if it happens in timewe only go through and look at everyone's vital signs about twice a dayit can be very helpful.
Theme 2: The EWS provides less‐experienced nurses with helpful age‐based reference ranges for vital signs that they use when caring for hospitalized children
Although this theme did not appear among physicians, nurses frequently noted that they referred to the scoring sheet as a reference for vital signs appropriate for hospitalized children. A surgical nurse with <1 year of experience said, In nursing school, I mostly dealt with adults. So, to figure out the different ranges for normal vital signs, it helps to have it listed on paper so I can see, 'Oh, I didn't realize that this 10‐year‐old's heart rate is higher than it should be.' A medical nurse with 14 years of experience cited the benefits for less‐experienced nurses, noting, [The EWS helps] newer nurses who don't know the ranges. Where it's Oh, my kid's blood pressure is 81 [mm Hg] over something, then they can look at their age and say, Oh, that is completely normal for a 2‐month‐old. But [before the EWS] there was nowhere to look to see the ranges. Unless you were [Pediatric Advanced Life Support] certified where you would know that stuff, there was a lot of anxiety related to vital signs.
Theme 3: The EWS provides concrete evidence of clinical changes in the form of a score. This empowers nurses to overcome escalation barriers and communicate their concerns, helping them take action to rescue their deteriorating patients
Nurses and physicians often described the role of the EWS as a source of objective evidence that a patient was exhibiting a concerning change. They shared the ways in which the EWS was used to convey concerns, noting most commonly that this was used as a communication tool by nurses to raise their concerns with physicians. A medical nurse with 23 years of experience said, [With the EWS] you feel like you have concrete evidence. It's not just a feeling [that] they are not looking as well as they were it feels scientific. Building upon this concept, a medical attending physician in practice for 2 years said, The EWS is a number that certainly gives people a sense of Here's the data behind why I am really coming to you and insisting on this. It is not calling and saying, I just have a bad feeling, it is, I have a bad feeling and his EWS has gone to a 9.
Theme 4: In some patients, the EWS may not help with decision making. These include patients who are very stable and have a low likelihood of deterioration, patients with abnormal physiology at baseline who consistently have very high EWSs, and patients experiencing neurologic deterioration
Nurses and physicians described some patient scenarios in which the EWS may not help with decision making. Discussing postoperative patients, a surgical nurse with 1 year of experience said, I love doing [the EWS] for some patients. I think it makes perfect sense. Then there are some patients [for whom] I am doing it just to do it because they are only here for 24 hours. They are completely stable. They never had 1 vital sign that was even a little bit off. It's kind of like we are just filling it out to fill it out. Commenting on patients at the other end of the spectrum, a medical attending physician in practice for 2 years said, [The EWS] can be a useful composite tool, but for specialty patients with abnormal baselines, I think it is much more a question of making sure you pay attention to the specific changes, whether it is the EWS or heart rate or vital signs or pain score or any of those things. A final area in which nurses and physicians identified weaknesses in the EWS surrounded neurologic deterioration. Specifically, nurses and physicians described experiences when the EWS increased minimally or not at all in patients with sudden seizures or concerning mental status changes that warranted escalation of care.
DISCUSSION
This study is the first to analyze viewpoints on the mechanisms by which EWSs impact decision making among physicians and nurses who had recently experienced score failures. Our study, performed in a children's hospital, builds upon the findings of related studies performed in hospitals that care primarily for adults.[23, 24, 25, 26, 27, 28] Andrews and Waterman found that nurses consider the utility of EWSs to extend beyond detecting deterioration by providing quantifiable evidence, packaged in the form of a score that improves communication between nurses and physicians.[23] Mackintosh and colleagues found that a RRS that included an EWS helped to formalize the way nurses and physicians understand deterioration, enable them to overcome hierarchical boundaries through structured discussions, and empower them to call for help.[24] In a quasi‐experimental study, McDonnell and colleagues found that an EWS improved self‐assessed knowledge, skills, and confidence of nursing staff to detect and manage deteriorating patients.[25] In addition, we describe novel findings, including the use of EWS parameters as reference ranges independent of the score, and specific situations when the EWS fails to support decision making. The weaknesses we identified could be used to drive EWS optimization for low‐risk patients who are stable as well as higher‐risk patients with abnormal baseline physiology and those at risk of neurologic deterioration.
This study has several limitations. Although the interviewers were not involved in RRS operations, it is possible that social desirability bias influenced responses. Next, we identified a knowledge gap among surgeons, and they contributed minimally to our findings. This is most likely because (1) surgical patients deteriorate on the wards less often than medical patients in our hospital, so surgeons are rarely presented with EWSs; (2) surgeons spend less time on the wards compared with medical physicians; and (3) surgical residents rotate in short blocks interspersed with rotations at other hospitals and may be less engaged in hospital safety initiatives.
CONCLUSIONS
Although EWSs perform only marginally well as statistical tools to predict clinical deterioration, nurses and physicians who recently experienced score failures described substantial benefits in using them to help identify deteriorating patients and transcend barriers to escalation of care by serving as objective communication tools. Combining an EWS with a clinician's judgment may result in a system better equipped to respond to deterioration than previous EWS studies focused on their test characteristics alone suggest. Future research should seek to compare and prospectively evaluate the clinical effectiveness of EWSs in real‐world settings.
Acknowledgments
Disclosures: This project was funded by the Pennsylvania Health Research Formula Fund Award (awarded to Keren and Bonafide) and the CHOP Nursing Research and Evidence‐Based Practice Award (awarded to Roberts). The funders did not influence the study design; the collection, analysis, or interpretation of data; the writing of the report; or the decision to submit the article for publication. The authors have no other conflicts to report.
- Institute for Healthcare Improvement. Overview of the Institute for Healthcare Improvement Five Million Lives Campaign. Available at: http://www.ihi.org/offerings/Initiatives/PastStrategicInitiatives/5MillionLivesCampaign/Pages/default.aspx. Accessed June 21, 2012.
- UK National Institute for Health and Clinical Excellence (NICE). Acutely Ill Patients in Hospital: Recognition of and Response to Acute Illness in Adults in Hospital. Available at: http://publications.nice.org.uk/acutely‐ill‐patients‐in‐hospital‐cg50. Published July 2007. Accessed June 21, 2012.
- , , , et al. Findings of the first consensus conference on medical emergency teams. Crit Care Med. 2006; 34(9):2463–2478.
- . Detecting and managing deterioration in children. Paediatr Nurs. 2005;17(1):32–35.
- , , , , , . Sensitivity of the Pediatric Early Warning Score to identify patient deterioration. Pediatrics. 2010;125(4):e763–e769.
- , , . Promoting care for acutely ill children—development and evaluation of a paediatric early warning tool. Intensive Crit Care Nurs. 2006;22(2):73–81.
- , , , . Prospective cohort study to test the predictability of the Cardiff and Vale paediatric early warning system. Arch Dis Child. 2009;94(8):602–606.
- , , . The Pediatric Early Warning System Score: a severity of illness score to predict urgent medical need in hospitalized children. J Crit Care. 2006;21(3):271–278.
- , , . Development and initial validation of the Bedside Paediatric Early Warning System score. Crit Care. 2009;13(4):R135.
- , , , et al. Multi‐centre validation of the Bedside Paediatric Early Warning System Score: a severity of illness score to detect evolving critical illness in hospitalized children. Crit Care. 2011;15(4):R184.
- , . Evaluation of a paediatric early warning tool—claims unsubstantiated. Intensive Crit Care Nurs. 2006;22(6):315–316.
- , , , et al. Development of a pragmatic measure for evaluating and optimizing rapid response systems. Pediatrics. 2012;129(4):e874–e881.
- , , , et al. Implementation of a multicenter rapid response system in pediatric academic hospitals is effective. Pediatrics. 2011;128(1):72–78.
- , , , , . Rapid response teams: a systematic review and meta‐analysis. Arch Intern Med. 2010;170(1):18–26.
- , , , et al. Implementation of a medical emergency team in a large pediatric teaching hospital prevents respiratory and cardiopulmonary arrests outside the intensive care unit. Pediatr Crit Care Med. 2007;8(3):236–246.
- , , , et al. Transition from a traditional code team to a medical emergency team and categorization of cardiopulmonary arrests in a children's center. Arch Pediatr Adolesc Med. 2008;162(2):117–122.
- , , , et al. Effect of a rapid response team on hospital‐wide mortality and code rates outside the ICU in a children's hospital. JAMA. 2007;298(19):2267–2274.
- , . Reduction of hospital mortality and of preventable cardiac arrest and death on introduction of a pediatric medical emergency team. Pediatr Crit Care Med. 2009;10(3):306–312.
- , , , et al. Implementation and impact of a rapid response team in a children's hospital. Jt Comm J Qual Patient Saf. 2007;33(7):418–425.
- , , . How many interviews are enough? An experiment with data saturation and variability. Field Methods. 2006;18(1):59–82.
- . Different approaches in grounded theory. In: Bryant A, Charmaz K, eds. The Sage Handbook of Grounded Theory. Los Angeles, CA: Sage; 2007:191–213.
- , . The Discovery of Grounded Theory: Strategies for Qualitative Research. New York, NY: Aldine De Gruyter; 1967.
- , . Packaging: a grounded theory of how to report physiological deterioration effectively. J Adv Nurs. 2005;52(5):473–481.
- , , . Understanding how rapid response systems may improve safety for the acutely ill patient: learning from the frontline. BMJ Qual Saf. 2012;21(2):135–144.
- , , , , , . A before and after study assessing the impact of a new model for recognizing and responding to early signs of deterioration in an acute hospital. J Adv Nurs. 2013;69(1):41–52.
- , . Overcoming gendered and professional hierarchies in order to facilitate escalation of care in emergency situations: the role of standardised communication protocols. Soc Sci Med. 2010;71(9):1683–1686.
- , , , , . Defining impact of a rapid response team: qualitative study with nurses, physicians and hospital administrators. BMJ Qual Saf. 2012;21(5):391–398.
- , , , , . Leading successful rapid response teams: a multisite implementation evaluation. J Nurs Adm. 2009;39(4):176–181.
- Institute for Healthcare Improvement. Overview of the Institute for Healthcare Improvement Five Million Lives Campaign. Available at: http://www.ihi.org/offerings/Initiatives/PastStrategicInitiatives/5MillionLivesCampaign/Pages/default.aspx. Accessed June 21, 2012.
- UK National Institute for Health and Clinical Excellence (NICE). Acutely Ill Patients in Hospital: Recognition of and Response to Acute Illness in Adults in Hospital. Available at: http://publications.nice.org.uk/acutely‐ill‐patients‐in‐hospital‐cg50. Published July 2007. Accessed June 21, 2012.
- , , , et al. Findings of the first consensus conference on medical emergency teams. Crit Care Med. 2006; 34(9):2463–2478.
- . Detecting and managing deterioration in children. Paediatr Nurs. 2005;17(1):32–35.
- , , , , , . Sensitivity of the Pediatric Early Warning Score to identify patient deterioration. Pediatrics. 2010;125(4):e763–e769.
- , , . Promoting care for acutely ill children—development and evaluation of a paediatric early warning tool. Intensive Crit Care Nurs. 2006;22(2):73–81.
- , , , . Prospective cohort study to test the predictability of the Cardiff and Vale paediatric early warning system. Arch Dis Child. 2009;94(8):602–606.
- , , . The Pediatric Early Warning System Score: a severity of illness score to predict urgent medical need in hospitalized children. J Crit Care. 2006;21(3):271–278.
- , , . Development and initial validation of the Bedside Paediatric Early Warning System score. Crit Care. 2009;13(4):R135.
- , , , et al. Multi‐centre validation of the Bedside Paediatric Early Warning System Score: a severity of illness score to detect evolving critical illness in hospitalized children. Crit Care. 2011;15(4):R184.
- , . Evaluation of a paediatric early warning tool—claims unsubstantiated. Intensive Crit Care Nurs. 2006;22(6):315–316.
- , , , et al. Development of a pragmatic measure for evaluating and optimizing rapid response systems. Pediatrics. 2012;129(4):e874–e881.
- , , , et al. Implementation of a multicenter rapid response system in pediatric academic hospitals is effective. Pediatrics. 2011;128(1):72–78.
- , , , , . Rapid response teams: a systematic review and meta‐analysis. Arch Intern Med. 2010;170(1):18–26.
- , , , et al. Implementation of a medical emergency team in a large pediatric teaching hospital prevents respiratory and cardiopulmonary arrests outside the intensive care unit. Pediatr Crit Care Med. 2007;8(3):236–246.
- , , , et al. Transition from a traditional code team to a medical emergency team and categorization of cardiopulmonary arrests in a children's center. Arch Pediatr Adolesc Med. 2008;162(2):117–122.
- , , , et al. Effect of a rapid response team on hospital‐wide mortality and code rates outside the ICU in a children's hospital. JAMA. 2007;298(19):2267–2274.
- , . Reduction of hospital mortality and of preventable cardiac arrest and death on introduction of a pediatric medical emergency team. Pediatr Crit Care Med. 2009;10(3):306–312.
- , , , et al. Implementation and impact of a rapid response team in a children's hospital. Jt Comm J Qual Patient Saf. 2007;33(7):418–425.
- , , . How many interviews are enough? An experiment with data saturation and variability. Field Methods. 2006;18(1):59–82.
- . Different approaches in grounded theory. In: Bryant A, Charmaz K, eds. The Sage Handbook of Grounded Theory. Los Angeles, CA: Sage; 2007:191–213.
- , . The Discovery of Grounded Theory: Strategies for Qualitative Research. New York, NY: Aldine De Gruyter; 1967.
- , . Packaging: a grounded theory of how to report physiological deterioration effectively. J Adv Nurs. 2005;52(5):473–481.
- , , . Understanding how rapid response systems may improve safety for the acutely ill patient: learning from the frontline. BMJ Qual Saf. 2012;21(2):135–144.
- , , , , , . A before and after study assessing the impact of a new model for recognizing and responding to early signs of deterioration in an acute hospital. J Adv Nurs. 2013;69(1):41–52.
- , . Overcoming gendered and professional hierarchies in order to facilitate escalation of care in emergency situations: the role of standardised communication protocols. Soc Sci Med. 2010;71(9):1683–1686.
- , , , , . Defining impact of a rapid response team: qualitative study with nurses, physicians and hospital administrators. BMJ Qual Saf. 2012;21(5):391–398.
- , , , , . Leading successful rapid response teams: a multisite implementation evaluation. J Nurs Adm. 2009;39(4):176–181.
Copyright © 2013 Society of Hospital Medicine