John H. Holmes, PhD

Study Design

We performed a case‐crossover study of children who experienced clinical deterioration. An alternative to the matched case‐control design, the case‐crossover design involves longitudinal within‐subject comparisons exclusively of case subjects such that an individual serves as his or her own control. It is most effective when studying intermittent exposures that result in transient changes in the risk of an acute event,[15, 16, 17] making it appropriate for our study.

Using the case‐crossover design, we compared a discrete time period in close proximity to the deterioration event, called the hazard interval, with earlier time periods in the hospitalization, called the control intervals.[15, 16, 17] In our primary analysis (Figure 1B), we defined the durations of these intervals as follows: We first censored the 2 hours immediately preceding the clinical deterioration event (hours 0 to 2). We made this decision a priori to exclude medications used after deterioration was recognized and resuscitation had already begun. The 12‐hour period immediately preceding the censored interval was the hazard interval (hours 2 to 14). Each 12‐hour period immediately preceding the hazard interval was a control interval (hours 14 to 26, 26 to 38, 38 to 50, and 50 to 62). Depending on the child's length of stay prior to the deterioration event, each hazard interval had 14 control intervals for comparison. In sensitivity analysis, we altered the durations of these intervals (see below).

Figure 1

Study Setting and Participants

We performed this study among children age <18 years who experienced clinical deterioration between January 1, 2005, and December 31, 2008, after being hospitalized on a general medical or surgical unit at The Children's Hospital of Philadelphia for 24 hours. Clinical deterioration was a composite outcome defined as cardiopulmonary arrest (CPA), acute respiratory compromise (ARC), or urgent ICU transfer. Cardiopulmonary arrest events required either pulselessness or a pulse with inadequate perfusion treated with chest compressions and/or defibrillation. Acute respiratory compromise events required respiratory insufficiency treated with bag‐valve‐mask or invasive airway interventions. Urgent ICU transfers included 1 of the following outcomes in the 12 hours after transfer: death, CPA, intubation, initiation of noninvasive ventilation, or administration of a vasoactive medication infusion used for the treatment of shock. Time zero was the time of the CPA/ARC, or the time at which the child arrived in the ICU for urgent transfers. These subjects also served as the cases for a previously published case‐control study evaluating different risk factors for deterioration.[18] The institutional review board of The Children's Hospital of Philadelphia approved the study.

At the time of the study, the hospital did not have a formal RRS. An immediate‐response code‐blue team was available throughout the study period for emergencies occurring outside the ICU. Physicians could also page the pediatric ICU fellow to discuss patients who did not require immediate assistance from the code‐blue team but were clinically deteriorating. There were no established triggering criteria.

Medication Exposures

Intravenous (IV) medications administered in the 72 hours prior to clinical deterioration were considered the exposures of interest. Each medication was included in 1 therapeutic classes assigned in the hospital's formulary (Lexicomp, Hudson, OH).[19] In order to determine which therapeutic classes to evaluate, we performed a power calculation using the sampsi_mcc package for Stata 12 (StataCorp, College Station, TX). We estimated that we would have 3 matched control intervals per hazard interval. We found that, in order to detect a minimum odds ratio of 3.0 with 80% power, a therapeutic class had to be administered in 5% of control periods. All therapeutic classes meeting that requirement were included in the analysis and are listed in Table 1. (See lists of the individual medications comprising each class in the Supporting Information, Tables 124, in the online version of this article.)

Data Collection

Data were abstracted from the electronic medication administration record (Sunrise Clinical Manager; Allscripts, Chicago, IL) into a database. For each subject, we recorded the name and time of administration of each IV medication given in the 72 hours preceding deterioration, as well as demographic, event, and hospitalization characteristics.

Statistical Analysis

We used univariable conditional logistic regression to evaluate the association between each therapeutic class and the composite outcome of clinical deterioration in the primary analysis. Because cases serve as their own controls in the case‐crossover design, this method inherently adjusts for all subject‐specific time‐invariant confounding variables, such as patient demographics, disease, and hospital‐ward characteristics.[15]

Sensitivity Analysis

Our primary analysis used a 2‐hour censored interval and 12‐hour hazard and control intervals. Excluding the censored interval from analysis was a conservative approach that we chose because our goal was to identify therapeutic classes associated with deterioration during a phase in which adverse outcomes may be prevented with early intervention. In order to test whether our findings were stable across different lengths of censored, hazard, and control intervals, we performed a sensitivity analysis, also using conditional logistic regression, on all therapeutic classes that were significant (P<0.05) in primary analysis. In 6 iterations of the sensitivity analysis, we varied the length of the hazard and control intervals between 8 and 12 hours, and the length of the censored interval between 0 and 4 hours (Figure 1AF). In a seventh iteration, we used a variant of the primary analysis in which we censored the first control interval (Figure 1G).

Primary Analysis

A total of 141 hazard intervals and 487 control intervals were included in the primary analysis, the results of which are shown in Table 3. Among the antimicrobial therapeutic classes, glycopeptide antibiotics (vancomycin), anaerobic antibiotics, third‐generation and fourth‐generation cephalosporins, and aminoglycoside antibiotics were significant. All of the anti‐inflammatory therapeutic classes, including systemic corticosteroids, anti‐inflammatory agents, and adrenal corticosteroids, were significant. All of the sedatives, hypnotics, and antianxiety therapeutic classes, including sedatives, benzodiazepines, hypnotics, and antianxiety agents, were significant. Among the narcotic analgesic therapeutic classes, only 1 class, narcotic analgesics (full opioid agonists), was significant. None of the gastrointestinal therapeutic classes were significant. Among the classes classified as other, loop diuretics and antidotes to hypersensitivity reactions (diphenhydramine) were significant.

Sensitivity Analysis

Of the 14 classes that were significant in primary analysis, we carried 9 forward to sensitivity analysis. The 5 that were not carried forward overlapped substantially with other classes that were carried forward. The decision of which overlapping class to carry forward was based upon (1) parsimony and (2) clinical relevance. This is described briefly in the footnotes to Table 3 (see Supporting information in the online version of this article for a full description of this process). Figure 2 presents the odds ratios and their 95% confidence intervals for the sensitivity analysis of each therapeutic class that was significant in primary analysis. Loop diuretics remained significantly associated with deterioration in all 7 iterations. Glycopeptide antibiotics (vancomycin), third‐generation and fourth‐generation cephalosporins, systemic corticosteroids, and benzodiazepines were significant in 6. Anaerobic antibiotics and narcotic analgesics (full opioid agonists) were significant in 5, and aminoglycoside antibiotics and antidotes to hypersensitivity reactions (diphenhydramine) in 4.

Figure 2

Study Design

We performed a case‐crossover study of children who experienced clinical deterioration. An alternative to the matched case‐control design, the case‐crossover design involves longitudinal within‐subject comparisons exclusively of case subjects such that an individual serves as his or her own control. It is most effective when studying intermittent exposures that result in transient changes in the risk of an acute event,[15, 16, 17] making it appropriate for our study.

Using the case‐crossover design, we compared a discrete time period in close proximity to the deterioration event, called the hazard interval, with earlier time periods in the hospitalization, called the control intervals.[15, 16, 17] In our primary analysis (Figure 1B), we defined the durations of these intervals as follows: We first censored the 2 hours immediately preceding the clinical deterioration event (hours 0 to 2). We made this decision a priori to exclude medications used after deterioration was recognized and resuscitation had already begun. The 12‐hour period immediately preceding the censored interval was the hazard interval (hours 2 to 14). Each 12‐hour period immediately preceding the hazard interval was a control interval (hours 14 to 26, 26 to 38, 38 to 50, and 50 to 62). Depending on the child's length of stay prior to the deterioration event, each hazard interval had 14 control intervals for comparison. In sensitivity analysis, we altered the durations of these intervals (see below).

Figure 1

Study Setting and Participants

We performed this study among children age <18 years who experienced clinical deterioration between January 1, 2005, and December 31, 2008, after being hospitalized on a general medical or surgical unit at The Children's Hospital of Philadelphia for 24 hours. Clinical deterioration was a composite outcome defined as cardiopulmonary arrest (CPA), acute respiratory compromise (ARC), or urgent ICU transfer. Cardiopulmonary arrest events required either pulselessness or a pulse with inadequate perfusion treated with chest compressions and/or defibrillation. Acute respiratory compromise events required respiratory insufficiency treated with bag‐valve‐mask or invasive airway interventions. Urgent ICU transfers included 1 of the following outcomes in the 12 hours after transfer: death, CPA, intubation, initiation of noninvasive ventilation, or administration of a vasoactive medication infusion used for the treatment of shock. Time zero was the time of the CPA/ARC, or the time at which the child arrived in the ICU for urgent transfers. These subjects also served as the cases for a previously published case‐control study evaluating different risk factors for deterioration.[18] The institutional review board of The Children's Hospital of Philadelphia approved the study.

At the time of the study, the hospital did not have a formal RRS. An immediate‐response code‐blue team was available throughout the study period for emergencies occurring outside the ICU. Physicians could also page the pediatric ICU fellow to discuss patients who did not require immediate assistance from the code‐blue team but were clinically deteriorating. There were no established triggering criteria.

Medication Exposures

Intravenous (IV) medications administered in the 72 hours prior to clinical deterioration were considered the exposures of interest. Each medication was included in 1 therapeutic classes assigned in the hospital's formulary (Lexicomp, Hudson, OH).[19] In order to determine which therapeutic classes to evaluate, we performed a power calculation using the sampsi_mcc package for Stata 12 (StataCorp, College Station, TX). We estimated that we would have 3 matched control intervals per hazard interval. We found that, in order to detect a minimum odds ratio of 3.0 with 80% power, a therapeutic class had to be administered in 5% of control periods. All therapeutic classes meeting that requirement were included in the analysis and are listed in Table 1. (See lists of the individual medications comprising each class in the Supporting Information, Tables 124, in the online version of this article.)

Data Collection

Data were abstracted from the electronic medication administration record (Sunrise Clinical Manager; Allscripts, Chicago, IL) into a database. For each subject, we recorded the name and time of administration of each IV medication given in the 72 hours preceding deterioration, as well as demographic, event, and hospitalization characteristics.

Statistical Analysis

We used univariable conditional logistic regression to evaluate the association between each therapeutic class and the composite outcome of clinical deterioration in the primary analysis. Because cases serve as their own controls in the case‐crossover design, this method inherently adjusts for all subject‐specific time‐invariant confounding variables, such as patient demographics, disease, and hospital‐ward characteristics.[15]

Sensitivity Analysis

Our primary analysis used a 2‐hour censored interval and 12‐hour hazard and control intervals. Excluding the censored interval from analysis was a conservative approach that we chose because our goal was to identify therapeutic classes associated with deterioration during a phase in which adverse outcomes may be prevented with early intervention. In order to test whether our findings were stable across different lengths of censored, hazard, and control intervals, we performed a sensitivity analysis, also using conditional logistic regression, on all therapeutic classes that were significant (P<0.05) in primary analysis. In 6 iterations of the sensitivity analysis, we varied the length of the hazard and control intervals between 8 and 12 hours, and the length of the censored interval between 0 and 4 hours (Figure 1AF). In a seventh iteration, we used a variant of the primary analysis in which we censored the first control interval (Figure 1G).

Primary Analysis

A total of 141 hazard intervals and 487 control intervals were included in the primary analysis, the results of which are shown in Table 3. Among the antimicrobial therapeutic classes, glycopeptide antibiotics (vancomycin), anaerobic antibiotics, third‐generation and fourth‐generation cephalosporins, and aminoglycoside antibiotics were significant. All of the anti‐inflammatory therapeutic classes, including systemic corticosteroids, anti‐inflammatory agents, and adrenal corticosteroids, were significant. All of the sedatives, hypnotics, and antianxiety therapeutic classes, including sedatives, benzodiazepines, hypnotics, and antianxiety agents, were significant. Among the narcotic analgesic therapeutic classes, only 1 class, narcotic analgesics (full opioid agonists), was significant. None of the gastrointestinal therapeutic classes were significant. Among the classes classified as other, loop diuretics and antidotes to hypersensitivity reactions (diphenhydramine) were significant.

Sensitivity Analysis

Of the 14 classes that were significant in primary analysis, we carried 9 forward to sensitivity analysis. The 5 that were not carried forward overlapped substantially with other classes that were carried forward. The decision of which overlapping class to carry forward was based upon (1) parsimony and (2) clinical relevance. This is described briefly in the footnotes to Table 3 (see Supporting information in the online version of this article for a full description of this process). Figure 2 presents the odds ratios and their 95% confidence intervals for the sensitivity analysis of each therapeutic class that was significant in primary analysis. Loop diuretics remained significantly associated with deterioration in all 7 iterations. Glycopeptide antibiotics (vancomycin), third‐generation and fourth‐generation cephalosporins, systemic corticosteroids, and benzodiazepines were significant in 6. Anaerobic antibiotics and narcotic analgesics (full opioid agonists) were significant in 5, and aminoglycoside antibiotics and antidotes to hypersensitivity reactions (diphenhydramine) in 4.

Figure 2

Study Design

We performed a case‐crossover study of children who experienced clinical deterioration. An alternative to the matched case‐control design, the case‐crossover design involves longitudinal within‐subject comparisons exclusively of case subjects such that an individual serves as his or her own control. It is most effective when studying intermittent exposures that result in transient changes in the risk of an acute event,[15, 16, 17] making it appropriate for our study.

Using the case‐crossover design, we compared a discrete time period in close proximity to the deterioration event, called the hazard interval, with earlier time periods in the hospitalization, called the control intervals.[15, 16, 17] In our primary analysis (Figure 1B), we defined the durations of these intervals as follows: We first censored the 2 hours immediately preceding the clinical deterioration event (hours 0 to 2). We made this decision a priori to exclude medications used after deterioration was recognized and resuscitation had already begun. The 12‐hour period immediately preceding the censored interval was the hazard interval (hours 2 to 14). Each 12‐hour period immediately preceding the hazard interval was a control interval (hours 14 to 26, 26 to 38, 38 to 50, and 50 to 62). Depending on the child's length of stay prior to the deterioration event, each hazard interval had 14 control intervals for comparison. In sensitivity analysis, we altered the durations of these intervals (see below).

Figure 1

Study Setting and Participants

We performed this study among children age <18 years who experienced clinical deterioration between January 1, 2005, and December 31, 2008, after being hospitalized on a general medical or surgical unit at The Children's Hospital of Philadelphia for 24 hours. Clinical deterioration was a composite outcome defined as cardiopulmonary arrest (CPA), acute respiratory compromise (ARC), or urgent ICU transfer. Cardiopulmonary arrest events required either pulselessness or a pulse with inadequate perfusion treated with chest compressions and/or defibrillation. Acute respiratory compromise events required respiratory insufficiency treated with bag‐valve‐mask or invasive airway interventions. Urgent ICU transfers included 1 of the following outcomes in the 12 hours after transfer: death, CPA, intubation, initiation of noninvasive ventilation, or administration of a vasoactive medication infusion used for the treatment of shock. Time zero was the time of the CPA/ARC, or the time at which the child arrived in the ICU for urgent transfers. These subjects also served as the cases for a previously published case‐control study evaluating different risk factors for deterioration.[18] The institutional review board of The Children's Hospital of Philadelphia approved the study.

At the time of the study, the hospital did not have a formal RRS. An immediate‐response code‐blue team was available throughout the study period for emergencies occurring outside the ICU. Physicians could also page the pediatric ICU fellow to discuss patients who did not require immediate assistance from the code‐blue team but were clinically deteriorating. There were no established triggering criteria.

Medication Exposures

Intravenous (IV) medications administered in the 72 hours prior to clinical deterioration were considered the exposures of interest. Each medication was included in 1 therapeutic classes assigned in the hospital's formulary (Lexicomp, Hudson, OH).[19] In order to determine which therapeutic classes to evaluate, we performed a power calculation using the sampsi_mcc package for Stata 12 (StataCorp, College Station, TX). We estimated that we would have 3 matched control intervals per hazard interval. We found that, in order to detect a minimum odds ratio of 3.0 with 80% power, a therapeutic class had to be administered in 5% of control periods. All therapeutic classes meeting that requirement were included in the analysis and are listed in Table 1. (See lists of the individual medications comprising each class in the Supporting Information, Tables 124, in the online version of this article.)

Data Collection

Data were abstracted from the electronic medication administration record (Sunrise Clinical Manager; Allscripts, Chicago, IL) into a database. For each subject, we recorded the name and time of administration of each IV medication given in the 72 hours preceding deterioration, as well as demographic, event, and hospitalization characteristics.

Statistical Analysis

We used univariable conditional logistic regression to evaluate the association between each therapeutic class and the composite outcome of clinical deterioration in the primary analysis. Because cases serve as their own controls in the case‐crossover design, this method inherently adjusts for all subject‐specific time‐invariant confounding variables, such as patient demographics, disease, and hospital‐ward characteristics.[15]

Sensitivity Analysis

Our primary analysis used a 2‐hour censored interval and 12‐hour hazard and control intervals. Excluding the censored interval from analysis was a conservative approach that we chose because our goal was to identify therapeutic classes associated with deterioration during a phase in which adverse outcomes may be prevented with early intervention. In order to test whether our findings were stable across different lengths of censored, hazard, and control intervals, we performed a sensitivity analysis, also using conditional logistic regression, on all therapeutic classes that were significant (P<0.05) in primary analysis. In 6 iterations of the sensitivity analysis, we varied the length of the hazard and control intervals between 8 and 12 hours, and the length of the censored interval between 0 and 4 hours (Figure 1AF). In a seventh iteration, we used a variant of the primary analysis in which we censored the first control interval (Figure 1G).

Primary Analysis

A total of 141 hazard intervals and 487 control intervals were included in the primary analysis, the results of which are shown in Table 3. Among the antimicrobial therapeutic classes, glycopeptide antibiotics (vancomycin), anaerobic antibiotics, third‐generation and fourth‐generation cephalosporins, and aminoglycoside antibiotics were significant. All of the anti‐inflammatory therapeutic classes, including systemic corticosteroids, anti‐inflammatory agents, and adrenal corticosteroids, were significant. All of the sedatives, hypnotics, and antianxiety therapeutic classes, including sedatives, benzodiazepines, hypnotics, and antianxiety agents, were significant. Among the narcotic analgesic therapeutic classes, only 1 class, narcotic analgesics (full opioid agonists), was significant. None of the gastrointestinal therapeutic classes were significant. Among the classes classified as other, loop diuretics and antidotes to hypersensitivity reactions (diphenhydramine) were significant.

Sensitivity Analysis

Of the 14 classes that were significant in primary analysis, we carried 9 forward to sensitivity analysis. The 5 that were not carried forward overlapped substantially with other classes that were carried forward. The decision of which overlapping class to carry forward was based upon (1) parsimony and (2) clinical relevance. This is described briefly in the footnotes to Table 3 (see Supporting information in the online version of this article for a full description of this process). Figure 2 presents the odds ratios and their 95% confidence intervals for the sensitivity analysis of each therapeutic class that was significant in primary analysis. Loop diuretics remained significantly associated with deterioration in all 7 iterations. Glycopeptide antibiotics (vancomycin), third‐generation and fourth‐generation cephalosporins, systemic corticosteroids, and benzodiazepines were significant in 6. Anaerobic antibiotics and narcotic analgesics (full opioid agonists) were significant in 5, and aminoglycoside antibiotics and antidotes to hypersensitivity reactions (diphenhydramine) in 4.

Figure 2

Overview

As 1 component of a larger study, we conducted semistructured interviews with nurses and physicians at The Children's Hospital of Philadelphia (CHOP) between May and October 2011. In separate subprojects using the same participants, the larger study also aimed to identify residual barriers to calling for urgent assistance and assess the role of families in the recognition of deterioration and MET activation.

Setting

The Children's Hospital of Philadelphia is an urban, tertiary‐care pediatric hospital with 504 beds. Surgical patients hospitalized outside of ICUs are cared for by surgeons and surgical nurses without pediatrician co‐management. Implementation of a RRS was prompted by serious safety events in which clinical deterioration was either not recognized or was recognized and not escalated. Prior to RRS implementation, a code blue team could be activated for patients in immediate need of resuscitation, or, for less‐urgent needs, a pediatric ICU fellow could be paged by physicians for informal consults.

A multidisciplinary team developed and pilot‐tested the RRS, then implemented it hospital‐wide in February 2010. Representing an aspect of a multipronged approach to improve safety culture, the RRS consisted of (1) an EWS based upon Parshuram's Bedside Paediatric Early Warning System,[8, 9, 10] calculated by hand on a paper form (see online supplementary content) at the same frequency as vital signs (usually every 4 hours), and (2) a 30‐minute response MET available for activation by any clinician for any concern, 24 hours per day, 7 days per week. Escalation guidelines included a prompt to activate the MET for a score that increased to the red zone (9). For concerns that could not wait 30 minutes, any hospital employee could activate the immediate‐response code blue team.

Utilization of the RRS at CHOP is high, with 23 calls to the MET per day and a combined MET/code‐blue team call rate of 27.8 per 1000 admissions.[12] Previously reported pediatric call rates range from 2.8 to 44.0, with a median of 9.6 per 1000 admissions across 6 studies.[13, 14, 15, 16, 17, 18, 19] Since implementation, there has been a statistically significant net reduction in critical deterioration events (unpublished data).

Participants

We recruited nurses and physicians who had recently cared for children age 18 years on general medical or surgical wards with false‐negative or false‐positive EWSs (instances when the score failed to predict deterioration). Recruitment ceased when we reached thematic data saturation (a qualitative research term for the point at which no new themes emerge with additional interviews).[20]

Data Collection

Through a detailed review of the relevant literature and consultation with experts, we developed a semistructured interview guide (see online supplementary content) to elicit nurses' and physicians' viewpoints regarding the mechanisms by which they use EWSs to support their decision making.

Experienced qualitative research scientists (F.K.B. and J.H.H.) trained 2 study interviewers (B.P. and K.M.T.). In order to minimize social‐desirability bias, the interviewers were not clinicians and were not involved in RRS operations. Each interview was recorded, professionally transcribed, and imported into NVivo 8.0 software for analysis (QSR International, Melbourne, Australia).

Data Analysis

We coded the interviews inductively, without using a predetermined set of themes. This approach is known as grounded theory methodology.[21] Two team members coded each interview independently. They then reviewed their coding together and discussed discrepancies until reaching consensus. In weekly meetings while the interviews were ongoing, we compared newly collected data with themes that had previously emerged in order to guide further thematic development and refinement (the constant comparative method).[22] After all of the interviews were completed and consensus had been reached for each individual interview, the study team convened a series of additional meetings to further refine and finalize the themes.

Human Subjects

The CHOP Institutional Review Board approved this study. All participants provided written informed consent.

Participants

We recruited 27 nurses and 30 physicians before reaching thematic data saturation. Because surgical patients are underrepresented relative to medical patients among the population with false‐positive and false‐negative scores in our hospital, this included 3 randomly selected surgical nurses and 7 randomly selected surgical physicians recruited to ensure thematic data saturation for surgical settings. Characteristics of the participants are displayed in Table 2.

Thematic Analysis

We provide the final themes, associated subthemes, and representative quotations below, with additional supporting quotations in Table 3. Because CHOP's MET is named the Critical Assessment Team, the term CAT appears in some quotations.

Of note, after interviewing 9 surgeons, we found that they were not very familiar with the EWS and had little to say either positively or negatively about the system. For example, when asked what they thought about the EWS, a surgical intern said, I have no idea. I don't have enough experience with it. This is probably the first time that I ever had anybody telling me that the system is in place. Therefore, surgeons did not contribute meaningfully to the themes below.

Theme 1: The EWS facilitates patient safety by alerting nurses and physicians to concerning vital sign changes and prompting them to think critically about the possibility of deterioration

Nurses and physicians frequently discussed the direct role of the EWS in revealing changes consistent with early signs of deterioration. A medical nurse with <1 year of experience said, The higher the number gets, the more it sets off a red flag to you to kind of keep an eye on certain things. They are just as important as taking a set of vitals. When asked if the EWS had ever helped to identify deterioration, a medical attending physician in practice for 5 years said, I think sometimes we will blow off, so to speak, certain things, but when you look at the numbers and you see a big [EWS] change versus if you were [just] looking at individual vital signs, then yeah, I think it has made a difference.

Nurses and physicians also discussed the role of the EWSs in prompting them to closely examine individual vital signs and think critically about whether or not a patient is exhibiting early signs of deterioration. A surgical nurse with <1 year of experience said, Sometimes I feel like if you want things to be okay you can kind of write them off, but when you have to write [the EWS] down it kind of jogs you to think, maybe something is going on or maybe someone else needs to know about this. A medical senior resident commented, I think it has alerted me earlier to changes in vital signs that I might not necessarily have known. I think there are nurses that use it and they see that there is an elevation and they call you about it. Then it makes me go back and look through and see what their vital signs are and if it happens in timewe only go through and look at everyone's vital signs about twice a dayit can be very helpful.

Overview

As 1 component of a larger study, we conducted semistructured interviews with nurses and physicians at The Children's Hospital of Philadelphia (CHOP) between May and October 2011. In separate subprojects using the same participants, the larger study also aimed to identify residual barriers to calling for urgent assistance and assess the role of families in the recognition of deterioration and MET activation.

Setting

The Children's Hospital of Philadelphia is an urban, tertiary‐care pediatric hospital with 504 beds. Surgical patients hospitalized outside of ICUs are cared for by surgeons and surgical nurses without pediatrician co‐management. Implementation of a RRS was prompted by serious safety events in which clinical deterioration was either not recognized or was recognized and not escalated. Prior to RRS implementation, a code blue team could be activated for patients in immediate need of resuscitation, or, for less‐urgent needs, a pediatric ICU fellow could be paged by physicians for informal consults.

A multidisciplinary team developed and pilot‐tested the RRS, then implemented it hospital‐wide in February 2010. Representing an aspect of a multipronged approach to improve safety culture, the RRS consisted of (1) an EWS based upon Parshuram's Bedside Paediatric Early Warning System,[8, 9, 10] calculated by hand on a paper form (see online supplementary content) at the same frequency as vital signs (usually every 4 hours), and (2) a 30‐minute response MET available for activation by any clinician for any concern, 24 hours per day, 7 days per week. Escalation guidelines included a prompt to activate the MET for a score that increased to the red zone (9). For concerns that could not wait 30 minutes, any hospital employee could activate the immediate‐response code blue team.

Utilization of the RRS at CHOP is high, with 23 calls to the MET per day and a combined MET/code‐blue team call rate of 27.8 per 1000 admissions.[12] Previously reported pediatric call rates range from 2.8 to 44.0, with a median of 9.6 per 1000 admissions across 6 studies.[13, 14, 15, 16, 17, 18, 19] Since implementation, there has been a statistically significant net reduction in critical deterioration events (unpublished data).

Participants

We recruited nurses and physicians who had recently cared for children age 18 years on general medical or surgical wards with false‐negative or false‐positive EWSs (instances when the score failed to predict deterioration). Recruitment ceased when we reached thematic data saturation (a qualitative research term for the point at which no new themes emerge with additional interviews).[20]

Data Collection

Through a detailed review of the relevant literature and consultation with experts, we developed a semistructured interview guide (see online supplementary content) to elicit nurses' and physicians' viewpoints regarding the mechanisms by which they use EWSs to support their decision making.

Experienced qualitative research scientists (F.K.B. and J.H.H.) trained 2 study interviewers (B.P. and K.M.T.). In order to minimize social‐desirability bias, the interviewers were not clinicians and were not involved in RRS operations. Each interview was recorded, professionally transcribed, and imported into NVivo 8.0 software for analysis (QSR International, Melbourne, Australia).

Data Analysis

We coded the interviews inductively, without using a predetermined set of themes. This approach is known as grounded theory methodology.[21] Two team members coded each interview independently. They then reviewed their coding together and discussed discrepancies until reaching consensus. In weekly meetings while the interviews were ongoing, we compared newly collected data with themes that had previously emerged in order to guide further thematic development and refinement (the constant comparative method).[22] After all of the interviews were completed and consensus had been reached for each individual interview, the study team convened a series of additional meetings to further refine and finalize the themes.

Human Subjects

The CHOP Institutional Review Board approved this study. All participants provided written informed consent.

Participants

We recruited 27 nurses and 30 physicians before reaching thematic data saturation. Because surgical patients are underrepresented relative to medical patients among the population with false‐positive and false‐negative scores in our hospital, this included 3 randomly selected surgical nurses and 7 randomly selected surgical physicians recruited to ensure thematic data saturation for surgical settings. Characteristics of the participants are displayed in Table 2.

Thematic Analysis

We provide the final themes, associated subthemes, and representative quotations below, with additional supporting quotations in Table 3. Because CHOP's MET is named the Critical Assessment Team, the term CAT appears in some quotations.

Of note, after interviewing 9 surgeons, we found that they were not very familiar with the EWS and had little to say either positively or negatively about the system. For example, when asked what they thought about the EWS, a surgical intern said, I have no idea. I don't have enough experience with it. This is probably the first time that I ever had anybody telling me that the system is in place. Therefore, surgeons did not contribute meaningfully to the themes below.

Theme 1: The EWS facilitates patient safety by alerting nurses and physicians to concerning vital sign changes and prompting them to think critically about the possibility of deterioration

Nurses and physicians frequently discussed the direct role of the EWS in revealing changes consistent with early signs of deterioration. A medical nurse with <1 year of experience said, The higher the number gets, the more it sets off a red flag to you to kind of keep an eye on certain things. They are just as important as taking a set of vitals. When asked if the EWS had ever helped to identify deterioration, a medical attending physician in practice for 5 years said, I think sometimes we will blow off, so to speak, certain things, but when you look at the numbers and you see a big [EWS] change versus if you were [just] looking at individual vital signs, then yeah, I think it has made a difference.

Nurses and physicians also discussed the role of the EWSs in prompting them to closely examine individual vital signs and think critically about whether or not a patient is exhibiting early signs of deterioration. A surgical nurse with <1 year of experience said, Sometimes I feel like if you want things to be okay you can kind of write them off, but when you have to write [the EWS] down it kind of jogs you to think, maybe something is going on or maybe someone else needs to know about this. A medical senior resident commented, I think it has alerted me earlier to changes in vital signs that I might not necessarily have known. I think there are nurses that use it and they see that there is an elevation and they call you about it. Then it makes me go back and look through and see what their vital signs are and if it happens in timewe only go through and look at everyone's vital signs about twice a dayit can be very helpful.

Overview

As 1 component of a larger study, we conducted semistructured interviews with nurses and physicians at The Children's Hospital of Philadelphia (CHOP) between May and October 2011. In separate subprojects using the same participants, the larger study also aimed to identify residual barriers to calling for urgent assistance and assess the role of families in the recognition of deterioration and MET activation.

Setting

The Children's Hospital of Philadelphia is an urban, tertiary‐care pediatric hospital with 504 beds. Surgical patients hospitalized outside of ICUs are cared for by surgeons and surgical nurses without pediatrician co‐management. Implementation of a RRS was prompted by serious safety events in which clinical deterioration was either not recognized or was recognized and not escalated. Prior to RRS implementation, a code blue team could be activated for patients in immediate need of resuscitation, or, for less‐urgent needs, a pediatric ICU fellow could be paged by physicians for informal consults.

A multidisciplinary team developed and pilot‐tested the RRS, then implemented it hospital‐wide in February 2010. Representing an aspect of a multipronged approach to improve safety culture, the RRS consisted of (1) an EWS based upon Parshuram's Bedside Paediatric Early Warning System,[8, 9, 10] calculated by hand on a paper form (see online supplementary content) at the same frequency as vital signs (usually every 4 hours), and (2) a 30‐minute response MET available for activation by any clinician for any concern, 24 hours per day, 7 days per week. Escalation guidelines included a prompt to activate the MET for a score that increased to the red zone (9). For concerns that could not wait 30 minutes, any hospital employee could activate the immediate‐response code blue team.

Utilization of the RRS at CHOP is high, with 23 calls to the MET per day and a combined MET/code‐blue team call rate of 27.8 per 1000 admissions.[12] Previously reported pediatric call rates range from 2.8 to 44.0, with a median of 9.6 per 1000 admissions across 6 studies.[13, 14, 15, 16, 17, 18, 19] Since implementation, there has been a statistically significant net reduction in critical deterioration events (unpublished data).

Participants

We recruited nurses and physicians who had recently cared for children age 18 years on general medical or surgical wards with false‐negative or false‐positive EWSs (instances when the score failed to predict deterioration). Recruitment ceased when we reached thematic data saturation (a qualitative research term for the point at which no new themes emerge with additional interviews).[20]

Data Collection

Through a detailed review of the relevant literature and consultation with experts, we developed a semistructured interview guide (see online supplementary content) to elicit nurses' and physicians' viewpoints regarding the mechanisms by which they use EWSs to support their decision making.

Experienced qualitative research scientists (F.K.B. and J.H.H.) trained 2 study interviewers (B.P. and K.M.T.). In order to minimize social‐desirability bias, the interviewers were not clinicians and were not involved in RRS operations. Each interview was recorded, professionally transcribed, and imported into NVivo 8.0 software for analysis (QSR International, Melbourne, Australia).

Data Analysis

We coded the interviews inductively, without using a predetermined set of themes. This approach is known as grounded theory methodology.[21] Two team members coded each interview independently. They then reviewed their coding together and discussed discrepancies until reaching consensus. In weekly meetings while the interviews were ongoing, we compared newly collected data with themes that had previously emerged in order to guide further thematic development and refinement (the constant comparative method).[22] After all of the interviews were completed and consensus had been reached for each individual interview, the study team convened a series of additional meetings to further refine and finalize the themes.

Human Subjects

The CHOP Institutional Review Board approved this study. All participants provided written informed consent.

Participants

We recruited 27 nurses and 30 physicians before reaching thematic data saturation. Because surgical patients are underrepresented relative to medical patients among the population with false‐positive and false‐negative scores in our hospital, this included 3 randomly selected surgical nurses and 7 randomly selected surgical physicians recruited to ensure thematic data saturation for surgical settings. Characteristics of the participants are displayed in Table 2.

Thematic Analysis

We provide the final themes, associated subthemes, and representative quotations below, with additional supporting quotations in Table 3. Because CHOP's MET is named the Critical Assessment Team, the term CAT appears in some quotations.

Of note, after interviewing 9 surgeons, we found that they were not very familiar with the EWS and had little to say either positively or negatively about the system. For example, when asked what they thought about the EWS, a surgical intern said, I have no idea. I don't have enough experience with it. This is probably the first time that I ever had anybody telling me that the system is in place. Therefore, surgeons did not contribute meaningfully to the themes below.

Theme 1: The EWS facilitates patient safety by alerting nurses and physicians to concerning vital sign changes and prompting them to think critically about the possibility of deterioration

Nurses and physicians frequently discussed the direct role of the EWS in revealing changes consistent with early signs of deterioration. A medical nurse with <1 year of experience said, The higher the number gets, the more it sets off a red flag to you to kind of keep an eye on certain things. They are just as important as taking a set of vitals. When asked if the EWS had ever helped to identify deterioration, a medical attending physician in practice for 5 years said, I think sometimes we will blow off, so to speak, certain things, but when you look at the numbers and you see a big [EWS] change versus if you were [just] looking at individual vital signs, then yeah, I think it has made a difference.

Nurses and physicians also discussed the role of the EWSs in prompting them to closely examine individual vital signs and think critically about whether or not a patient is exhibiting early signs of deterioration. A surgical nurse with <1 year of experience said, Sometimes I feel like if you want things to be okay you can kind of write them off, but when you have to write [the EWS] down it kind of jogs you to think, maybe something is going on or maybe someone else needs to know about this. A medical senior resident commented, I think it has alerted me earlier to changes in vital signs that I might not necessarily have known. I think there are nurses that use it and they see that there is an elevation and they call you about it. Then it makes me go back and look through and see what their vital signs are and if it happens in timewe only go through and look at everyone's vital signs about twice a dayit can be very helpful.

Design

We performed a case‐control study among children, younger than 18 years old, hospitalized for >24 hours between January 1, 2005 and December 31, 2008. The case group consisted of children who experienced clinical deterioration, a composite outcome defined as cardiopulmonary arrest (CPA), acute respiratory compromise (ARC), or urgent ICU transfer, while on a non‐ICU unit. ICU transfers were considered urgent if they included at least one of the following outcomes in the 12 hours after transfer: death, CPA, intubation, initiation of noninvasive ventilation, or administration of a vasoactive medication infusion used for the treatment of shock. The control group consisted of a random sample of patients matched 3:1 to cases if they met the criteria of being on a non‐ICU unit at the same time as their matched case.

Variables and Measurements

We collected data on demographics, complex chronic conditions (CCCs), other patient characteristics, and laboratory studies. CCCs were specific diagnoses divided into the following 9 categories according to an established framework: neuromuscular, cardiovascular, respiratory, renal, gastrointestinal, hematologic/emmmunologic, metabolic, malignancy, and genetic/congenital defects.9 Other patient characteristics evaluated included age, weight‐for‐age, gestational age, history of transplant, time from hospital admission to event, recent ICU stays, administration of total parenteral nutrition, use of a patient‐controlled analgesia pump, and presence of medical devices including central venous lines and enteral tubes (naso‐gastric, gastrostomy, or jejunostomy).

Laboratory studies evaluated included hemoglobin value, white blood cell count, and blood culture drawn in the preceding 72 hours. We included these laboratory studies in this predictive score because we hypothesized that they represented factors that increased a child's risk of deterioration over time, as opposed to signs of acute deterioration that would be more appropriate for a detective score.

Statistical Analysis

We used conditional logistic regression for the bivariable and multivariable analyses to account for the matching. We derived the predictive score using an established method10 in which the regression coefficients for each covariate were divided by the smallest coefficient, and then rounded to the nearest integer, to establish each variable's sub‐score. We grouped the total scores into very low, low, intermediate, and high‐risk groups, calculated overall stratum‐specific likelihood ratios (SSLRs), and estimated stratum‐specific probabilities of deterioration for each group.

Patient Characteristics

We identified 12 CPAs, 41 ARCs, and 699 urgent ICU transfers during the study period. A total of 141 cases met our strict criteria for inclusion (see Figure in Supporting Online Appendix: Supplement to Methods Section in the online version of this article) among approximately 96,000 admissions during the study period, making the baseline incidence of events (pre‐test probability) approximately 0.15%. The case and control groups were similar in age, sex, and family‐reported race/ethnicity. Cases had been hospitalized longer than controls at the time of their event, were less likely to have been on a surgical service, and were less likely to survive to hospital discharge (Table 1). There was a high prevalence of CCCs among both cases and controls; 78% of cases and 52% of controls had at least 1 CCC.

Unadjusted (Bivariable) Analysis

Results of bivariable analysis are shown in Table 2.

Adjusted (Multivariable) Analysis

The multivariable conditional logistic regression model included 7 independent risk factors for deterioration (Table 3): age <1 year, epilepsy, congenital/genetic defects, history of transplant, enteral tubes, hemoglobin <10 g/dL, and blood culture drawn in the preceding 72 hours.

Predictive Score

The range of the resulting predictive score was 0 to 12. The median score among cases was 4, and the median score among controls was 1 (P < 0.001). The area under the receiver operating characteristic curve was 0.78 (95% confidence interval 0.74‐0.83).

We grouped the scores by SSLRs into 4 risk strata and calculated each group's estimated post‐test probability of deterioration based on the pre‐test probability of deterioration of 0.15% (Table 4). The very low‐risk group had a probability of deterioration of 0.06%, less than one‐half the pre‐test probability. The low‐risk group had a probability of deterioration of 0.18%, similar to the pre‐test probability. The intermediate‐risk group had a probability of deterioration of 0.39%, 2.6 times higher than the pre‐test probability. The high‐risk group had a probability of deterioration of 12.60%, 84 times higher than the pre‐test probability.

Design

We performed a case‐control study among children, younger than 18 years old, hospitalized for >24 hours between January 1, 2005 and December 31, 2008. The case group consisted of children who experienced clinical deterioration, a composite outcome defined as cardiopulmonary arrest (CPA), acute respiratory compromise (ARC), or urgent ICU transfer, while on a non‐ICU unit. ICU transfers were considered urgent if they included at least one of the following outcomes in the 12 hours after transfer: death, CPA, intubation, initiation of noninvasive ventilation, or administration of a vasoactive medication infusion used for the treatment of shock. The control group consisted of a random sample of patients matched 3:1 to cases if they met the criteria of being on a non‐ICU unit at the same time as their matched case.

Variables and Measurements

We collected data on demographics, complex chronic conditions (CCCs), other patient characteristics, and laboratory studies. CCCs were specific diagnoses divided into the following 9 categories according to an established framework: neuromuscular, cardiovascular, respiratory, renal, gastrointestinal, hematologic/emmmunologic, metabolic, malignancy, and genetic/congenital defects.9 Other patient characteristics evaluated included age, weight‐for‐age, gestational age, history of transplant, time from hospital admission to event, recent ICU stays, administration of total parenteral nutrition, use of a patient‐controlled analgesia pump, and presence of medical devices including central venous lines and enteral tubes (naso‐gastric, gastrostomy, or jejunostomy).

Laboratory studies evaluated included hemoglobin value, white blood cell count, and blood culture drawn in the preceding 72 hours. We included these laboratory studies in this predictive score because we hypothesized that they represented factors that increased a child's risk of deterioration over time, as opposed to signs of acute deterioration that would be more appropriate for a detective score.

Statistical Analysis

We used conditional logistic regression for the bivariable and multivariable analyses to account for the matching. We derived the predictive score using an established method10 in which the regression coefficients for each covariate were divided by the smallest coefficient, and then rounded to the nearest integer, to establish each variable's sub‐score. We grouped the total scores into very low, low, intermediate, and high‐risk groups, calculated overall stratum‐specific likelihood ratios (SSLRs), and estimated stratum‐specific probabilities of deterioration for each group.

Patient Characteristics

We identified 12 CPAs, 41 ARCs, and 699 urgent ICU transfers during the study period. A total of 141 cases met our strict criteria for inclusion (see Figure in Supporting Online Appendix: Supplement to Methods Section in the online version of this article) among approximately 96,000 admissions during the study period, making the baseline incidence of events (pre‐test probability) approximately 0.15%. The case and control groups were similar in age, sex, and family‐reported race/ethnicity. Cases had been hospitalized longer than controls at the time of their event, were less likely to have been on a surgical service, and were less likely to survive to hospital discharge (Table 1). There was a high prevalence of CCCs among both cases and controls; 78% of cases and 52% of controls had at least 1 CCC.

Unadjusted (Bivariable) Analysis

Results of bivariable analysis are shown in Table 2.

Adjusted (Multivariable) Analysis

The multivariable conditional logistic regression model included 7 independent risk factors for deterioration (Table 3): age <1 year, epilepsy, congenital/genetic defects, history of transplant, enteral tubes, hemoglobin <10 g/dL, and blood culture drawn in the preceding 72 hours.

Predictive Score

The range of the resulting predictive score was 0 to 12. The median score among cases was 4, and the median score among controls was 1 (P < 0.001). The area under the receiver operating characteristic curve was 0.78 (95% confidence interval 0.74‐0.83).

We grouped the scores by SSLRs into 4 risk strata and calculated each group's estimated post‐test probability of deterioration based on the pre‐test probability of deterioration of 0.15% (Table 4). The very low‐risk group had a probability of deterioration of 0.06%, less than one‐half the pre‐test probability. The low‐risk group had a probability of deterioration of 0.18%, similar to the pre‐test probability. The intermediate‐risk group had a probability of deterioration of 0.39%, 2.6 times higher than the pre‐test probability. The high‐risk group had a probability of deterioration of 12.60%, 84 times higher than the pre‐test probability.

Design

We performed a case‐control study among children, younger than 18 years old, hospitalized for >24 hours between January 1, 2005 and December 31, 2008. The case group consisted of children who experienced clinical deterioration, a composite outcome defined as cardiopulmonary arrest (CPA), acute respiratory compromise (ARC), or urgent ICU transfer, while on a non‐ICU unit. ICU transfers were considered urgent if they included at least one of the following outcomes in the 12 hours after transfer: death, CPA, intubation, initiation of noninvasive ventilation, or administration of a vasoactive medication infusion used for the treatment of shock. The control group consisted of a random sample of patients matched 3:1 to cases if they met the criteria of being on a non‐ICU unit at the same time as their matched case.

Variables and Measurements

We collected data on demographics, complex chronic conditions (CCCs), other patient characteristics, and laboratory studies. CCCs were specific diagnoses divided into the following 9 categories according to an established framework: neuromuscular, cardiovascular, respiratory, renal, gastrointestinal, hematologic/emmmunologic, metabolic, malignancy, and genetic/congenital defects.9 Other patient characteristics evaluated included age, weight‐for‐age, gestational age, history of transplant, time from hospital admission to event, recent ICU stays, administration of total parenteral nutrition, use of a patient‐controlled analgesia pump, and presence of medical devices including central venous lines and enteral tubes (naso‐gastric, gastrostomy, or jejunostomy).

Laboratory studies evaluated included hemoglobin value, white blood cell count, and blood culture drawn in the preceding 72 hours. We included these laboratory studies in this predictive score because we hypothesized that they represented factors that increased a child's risk of deterioration over time, as opposed to signs of acute deterioration that would be more appropriate for a detective score.

Statistical Analysis

We used conditional logistic regression for the bivariable and multivariable analyses to account for the matching. We derived the predictive score using an established method10 in which the regression coefficients for each covariate were divided by the smallest coefficient, and then rounded to the nearest integer, to establish each variable's sub‐score. We grouped the total scores into very low, low, intermediate, and high‐risk groups, calculated overall stratum‐specific likelihood ratios (SSLRs), and estimated stratum‐specific probabilities of deterioration for each group.

Patient Characteristics

We identified 12 CPAs, 41 ARCs, and 699 urgent ICU transfers during the study period. A total of 141 cases met our strict criteria for inclusion (see Figure in Supporting Online Appendix: Supplement to Methods Section in the online version of this article) among approximately 96,000 admissions during the study period, making the baseline incidence of events (pre‐test probability) approximately 0.15%. The case and control groups were similar in age, sex, and family‐reported race/ethnicity. Cases had been hospitalized longer than controls at the time of their event, were less likely to have been on a surgical service, and were less likely to survive to hospital discharge (Table 1). There was a high prevalence of CCCs among both cases and controls; 78% of cases and 52% of controls had at least 1 CCC.

Unadjusted (Bivariable) Analysis

Results of bivariable analysis are shown in Table 2.

Adjusted (Multivariable) Analysis

The multivariable conditional logistic regression model included 7 independent risk factors for deterioration (Table 3): age <1 year, epilepsy, congenital/genetic defects, history of transplant, enteral tubes, hemoglobin <10 g/dL, and blood culture drawn in the preceding 72 hours.

Predictive Score

The range of the resulting predictive score was 0 to 12. The median score among cases was 4, and the median score among controls was 1 (P < 0.001). The area under the receiver operating characteristic curve was 0.78 (95% confidence interval 0.74‐0.83).

We grouped the scores by SSLRs into 4 risk strata and calculated each group's estimated post‐test probability of deterioration based on the pre‐test probability of deterioration of 0.15% (Table 4). The very low‐risk group had a probability of deterioration of 0.06%, less than one‐half the pre‐test probability. The low‐risk group had a probability of deterioration of 0.18%, similar to the pre‐test probability. The intermediate‐risk group had a probability of deterioration of 0.39%, 2.6 times higher than the pre‐test probability. The high‐risk group had a probability of deterioration of 12.60%, 84 times higher than the pre‐test probability.