Affiliations
Geriatric Research, Education, and Clinical Center (GRECC) and Center for Health Services Research, Veterans Affairs Tennessee Valley Healthcare System, Nashville, Tennessee
Department of Biostatistics, Vanderbilt University School of Medicine, Nashville, Tennessee
Given name(s)
Daniel J.
Family name
France
Degrees
PhD, MPH

Comparing Collaborative and Toolkit QI

Article Type
Changed
Mon, 05/22/2017 - 21:44
Display Headline
Quality improvement projects targeting health care–associated infections: Comparing virtual collaborative and toolkit approaches

Continuous quality improvement (CQI) methodologies provide a framework for initiating and sustaining improvements in complex systems.1 By definition, CQI engages frontline staff in iterative problem solving using plandostudyact cycles of learning, with decision‐making based on real‐time process measurements.2 The Institute for Healthcare Improvement (IHI) has sponsored Breakthrough Series Collaboratives since 1996 to accelerate the uptake and impact of quality improvement (QI).3, 4 These collaboratives are typically guided by evidence‐based clinical practice guidelines, incorporate change methodologies, and rely on clinical and process improvement subject matter experts. Through the collaborative network, teams share knowledge and ideas about effective and ineffective interventions as well as strategies for overcoming barriers. The collaborative curriculum includes CQI methodology, multidisciplinary teamwork, leadership support, and tools for measurement. Participants are typically required to invest resources and send teams to face‐to‐face goal‐oriented meetings. It is costly for a large healthcare organization to incorporate travel to a learning session conference into its collaborative model. Thus, we attempted virtual learning sessions that make use of webcasts, a Web site, and teleconference calls for tools and networking.5

A recent derivative of collaboratives has been deployment of toolkits for QI. Intuition suggests that such toolkits may help to enable change, and thus some agencies advocate the simpler approach of disseminating toolkits as a change strategy.6 Toolkit dissemination is a passive approach in contrast to collaborative participation, and its effectiveness has not been critically examined in evidence‐based literature.

We sought to compare the virtual collaborative model with the toolkit model for improving care. Recommendations and guidelines for central lineassociated bloodstream infection (CLABSI) and ventilator‐associated pneumonia (VAP) prevention have not been implemented reliably, resulting in unnecessary intensive care unit (ICU) morbidity and mortality and fostering a national call for improvement.7 Our aim was to compare the effectiveness of the virtual collaborative and toolkit approaches on preventing CLABSI and VAP in the ICU.

Methods

This cluster randomized trial included medical centers within the Hospital Corporation of America (HCA), a network of hospitals located primarily in the southern United States. To minimize contamination bias between study groups within the same facility, the unit of randomization was the hospital and implementation was at the level of the ICU. The project received approval from the Vanderbilt University Institutional Review Board.

Leaders of all medical centers with at least 1 adult or pediatric ICU received an invitation from HCA leadership to participate in a QI initiative. Facility clinicians and managers completed baseline surveys (shown in the Supporting Information) on hospital characteristics, types of ICUs, patient safety climate, and QI resources between July and November 2005. Hospital‐level data were extracted from the enterprise‐wide data warehouse. Hospitals willing to participate were matched on geographic location and ICU volume and then randomized into either the Virtual Collaborative (n = 31) or Toolkit (n = 30) groups in December 20058; 1 of the hospitals was sold, yielding 29 hospitals in the Toolkit (n = 29) group. The study lasted 18 months from January 2006 through September 2007, with health careassociated infection data collected through December 2007, and follow‐up data collection through April 2008.

The QI initiative included educational opportunities, evidence‐based clinical prevention interventions, and processes and tools to implement and measure the impact of these interventions. Participants in both groups were offered interactive Web seminars during the study period; 5 of these seminars were on clinical subject matter, and 5 seminars were on patient safety, charting use of statistical process control and QI methods. The interventions were evidence‐based care bundles.9 The key interventions for preventing CLABSI were routine hand hygiene, use of chlorhexidine skin antisepsis, maximal barrier precautions during catheter insertion, catheter site and care, and avoidance of routine replacement of catheters. The key interventions to prevent VAP were routine elevation of head of the bed, regular oral care, daily sedation vacations, daily assessment of readiness to extubate, secretion cleaning, peptic ulcer disease prophylaxis, and deep vein thrombosis prophylaxis.

Toolkit Group

Hospitals randomized to this arm received a toolkit during study month 1 containing a set of evidence‐based guidelines and fact sheets for preventing CLABSI and VAP, a review of QI and teamwork methods, standardized data collection tools, and standardized charting tools. The nurse and quality managers for the Toolkit ICUs were provided ad libitum access to the HCA intranet toolkit Web site containing all of the educational seminars, clinical tools, and QI tools. Otherwise, ICUs in this group were on their own to initiate and implement a local hospital QI initiative to prevent CLABSI and VAP.

Virtual Collaborative Group

In addition to the materials and Web site support described above, facility leaders and managers in this Virtual Collaborative group agreed to participate in a virtual collaborative to develop processes to more reliably implement evidence‐based interventions to prevent CLABSI and VAP. The collaboration differed from the Breakthrough Series model3, 4 in that teams did not come together for face‐to‐face educational and planning sessions but instead attended Web seminars and teleconferences for reporting back to the larger group.5 Teams were supported through monthly educational and troubleshooting conference calls, individual coaching coordinated by the HCA corporate office of quality, safety, and performance improvement, and an e‐mail listserv designed to stimulate interaction among teams.

Clinical Outcome Measures

Although most participating hospitals defined CLABSI and VAP using the Centers for Disease Control and Prevention definitions, data collection and surveillance methods varied across hospitals.10 Education was provided to standardize outcome measurement. A data registry Web application was created as a new tool for infection control data entry, and healthcare‐associated infection data reporting by the infection control personnel was mandated starting the first quarter of 2006. To verify electronic data and correct missing information, the infection control personnel were requested to complete a retrospective data collection sheet providing quarterly reports from January 2005 through December 2007 on ICU infection events as well as total catheter days and ventilator days to allow calculation of event rates. Outcome measures of CLABSI and VAP were at the level of the hospital.

Follow‐Up

The HCA e‐mail distribution and collection routine was employed for the follow‐up survey of ICU nurse and quality managers for all participating medical centers from January 2008 through April 2008. A single survey (shown in the Supporting Information) was requested from each participating ICU. The ICU‐level surveys included questions about the implementation of the CLABSI and VAP process interventions, access of tools, participation in Web seminars, and use of QI strategies.11, 12 The postintervention survey also assessed the character and amount of implementation and teamwork activity expended.

Median CLABSI and VAP rates for a 3‐month baseline and quarterly postintervention periods were compared between the 2 study groups. The CLABSI and VAP infection rates were also analyzed using hierarchical negative binomial regression models to model infection rate changes over time (time in months and group by time interaction effects) and account for clustering of ICUs within hospitals and adjusting for baseline covariates. Baseline and process variables at the hospital and ICU level were compared using chi‐square tests and t tests according to the type of measurement. Time‐to‐event analyses were conducted to compare the groups on time to initiation of a care process. All analyses were conducted using the (R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2010).

The power of the study was calculated a priori with a 1‐tailed alpha of 0.05 and group size of 30. We hypothesized a 50% decrease in hospital‐associated infection rates for the Collaborative group vs. a 10% to 15% decrease for the Toolkit group. The calculations yielded power ranging from a low of 82% to a high of 91% for testing group differences.13

Results

Participating facilities included rural (11%), inner city (28%), and suburban (61%) medical centers. The 60 participating sites did not differ in administrative variables from the 113 nonparticipating HCA sites (results not shown). The median hospital size was 177 beds and the median ICU size was 16 beds. The hospitals did not differ between study groups (Table 1). At baseline, 45% of the facilities reported having a CLABSI program and 62% a VAP program.

Baseline Characteristics of the Virtual Collaborative and Toolkit Groups
Hospital Factors at BaselineVirtual CollaborativeToolkitP Value
  • Abbreviations: IQR, interquartile range; SD, standard deviation.

  • One of the 30 hospitals randomized to the Toolkit group was subsequently sold, resulting in 29 hospitals for this study condition.

Number of hospitals3129* 
ICU annual patient volume, median (IQR)568 (294, 904)578 (244, 1077)0.93
ICU patient length of stay days, median (IQR)3882 (1758, 5718)4228 (1645, 6725)0.95
ICU mortality rate, percent (SD)5.7% (3.1%)7.1% (3.6%)0.13
Medicare/Medicaid, percent (SD)68.6% (9.5%)68.5% (10.1%)0.95
Percent admitted to ICU from the Emergency Department (SD)71% (15%)67% (20%)0.27
Percent female (SD)49.7% (5.7%)50.3% (7.7%)0.79
Medicare case‐mix weight, mean (SD)1221 (1007)1295 (1110)0.82
Percent hospitalist ICU management47%40%0.61

The baseline and quarterly median and pooled infection rates for the Toolkit and Collaboration groups are shown in Table 2 for CLABSI and in Table 3 for VAP. There were no significant differences in the baseline rates for either CLABSI (P = 0.24) or VAP (P = 0.72) between the Collaborative and Toolkit groups. There was no significant change for either CLABSI or VAP outcomes at either 12 or 18 months of follow‐up. The median bloodstream infection rate for all participating hospitals was 2.27 at baseline, 1.18 at 12 months (P = 0.13), and 2.23 per 1000 catheter days 18 months later (P = 0.95). The median VAP rate for participating hospitals was 2.90 at baseline, 2.67 at 12 months (P = 0.44), and 2.52 per 1000 ventilator days 18 months later (P = 0.84). The hierarchical regression analysis found that neither the Collaborative nor Toolkit groups improved CLABSI (P = 0.75 and P = 0.83, respectively) or VAP (P = 0.61 and P = 0.37, respectively) rates over time, and there was no differential performance between the 2 groups for either outcome (bloodstream infection, P = 0.71; VAP, P = 0.80).

CLABSI Rates, per 1000 Catheter Days, Overall and by Study Group
 OverallVirtual CollaborativeToolkit
 N = 59 HospitalsN = 30 HospitalsN = 29 Hospitals
Study PeriodHospital Median (IQR)Rate Pooled Across HospitalsHospital Median (IQR)Rate Pooled Across HospitalsHospital Median (IQR)Rate Pooled Across Hospitals
  • Abbreviation: IQR, interquartile range.

Baseline2.27 (0.00‐3.98)2.421.84 (0.00‐3.83)1.672.42 (0.65‐6.80)3.05
3 Month2.27 (1.30‐4.69)2.612.24 (0.54‐4.69)2.342.47 (1.48‐5.35)2.85
6 Month2.37 (0.00‐4.29)2.732.28 (0.00‐3.73)2.352.54 (0.00‐4.98)3.09
9 Month1.66 (0.00‐3.84)2.451.76 (0.00‐3.74)2.281.23 (0.00‐3.93)2.59
12 Month1.18 (0.00‐3.10)2.171.18 (0.00‐2.71)1.721.17 (0.00‐3.61)2.58
15 Month1.93 (0.00‐4.25)2.292.04 (0.00‐4.91)2.531.77 (0.00‐3.30)2.08
18 Month2.23 (0.00‐4.97)2.732.76 (0.00‐4.67)2.751.16 (0.00‐5.46)2.72
VAP Rates per 1000 Ventilator Days, Overall and by Study Group
Study PeriodOverallVirtual CollaborativeToolkit
N = 59 HospitalsN = 30 HospitalsN = 29 Hospitals
Hospital Median (IQR)Rate Pooled Across HospitalsHospital Median (IQR)Rate Pooled Across HospitalsHospital Median (IQR)Rate Pooled Across Hospitals
  • Abbreviation: IQR, interquartile range.

Baseline2.90 (0.00‐6.14)3.972.14 (0.00‐6.09)3.433.49 (0.00‐7.04)4.36
3 Month3.12 (0.00‐8.40)4.463.01 (0.00‐9.11)4.223.32 (0.00‐8.25)4.62
6 Month3.40 (0.00‐7.53)4.972.72 (0.00‐7.09)4.814.61 (0.00‐9.37)5.10
9 Month1.49 (0.00‐4.87)2.990 (0.00‐3.94)2.512.27 (0.00‐6.27)3.36
12 Month2.67 (0.00‐4.60)4.392.67 (0.00‐4.47)3.822.66 (0.00‐4.82)4.95
15 Month3.06 (0.00‐5.10)4.032.40 (0.00‐3.94)3.573.65 (1.15‐6.57)4.45
18 Month2.52 (0.00‐7.45)4.612.93 (0.00‐7.63)5.022.06 (0.00‐6.59)4.31

The poststudy survey was completed by 27 of 31 (87%) of Collaborative group hospitals and 19 of the 29 (66%) Toolkit hospitals. Both groups reported QI improvement efforts to prevent CLABSI (Collaborative 97% vs. Toolkit 88%, P = 0.29) and VAP (Collaborative 97% vs. Toolkit 96%, P = 0.99). Eighty‐three percent of the Collaborative group implemented all components of the bloodstream infection prevention interventions compared with 64% for the Toolkit group (P = 0.13; Figure 1). The Collaborative group implemented daily catheter review more often than the Toolkit group (P = 0.04) and began the process implementation sooner following study implementation (P = 0.006 vs. Toolkit; see Supporting Information Figure). Eighty‐six percent of the Collaborative group implemented the complete VAP prevention interventions vs. 64% of the Toolkit group (P = 0.06; Figure 1) and the Collaborative group conducted the sedation vacation intervention more often (P = 0.03).

Figure 1
(A) Follow‐up survey of self‐reported implementation of key CLABSI prevention interventions by study group. (B) Follow‐up survey of self‐reported implementation of key VAP prevention interventions by study group.

The Collaborative group participated in 57% of the seminars, whereas the Toolkit group participated in 39% (P = 0.014). Members of both groups attended more than half the clinical topics (Collaborative 64% vs. Toolkit 56%, P = 0.37). The Collaborative group had greater participation in the data and method topics (Collaborative 50% vs. Toolkit 22%, P < 0.001). The proportion of hospitals finding the seminars useful to their QI efforts was 49% for the Collaborative and 30% for the Toolkit group (P = 0.017). When restricted to hospitals that participated in the seminars, the usefulness rating was higher for both clinical (91% for the Collaborative and 86% for Toolkit) and Data/Methods (79% for Collaborative and 55% for Toolkit) topics.

A set of 14 tools were produced during the study period (Table 4); 9 clinically related tools (eg, checklists, algorithms, protocols, and flowsheets) and 5 data monitoring and quality improvement tools (eg, easy‐to‐use statistical process control spreadsheet templates, quality improvement tools, and computer tools). The Collaborative group downloaded a median of 10 tools and the Toolkit group a median of 7 (P = 0.051). The groups did not differ in their access to the clinical tools (P = 0.23) but the Collaborative group accessed a greater proportion of the data/methods tools (P = 0.004).

Follow‐up Survey on Study Groups' Tool Use and Strategies for Improvement
Tool Access and StrategiesCollaborative HospitalsaTool Kit HospitalsaP‐value
N = 36 ICUsN = 25 ICUs
  • Post‐survey respondents included 36 ICUs in 26 of the 30 Collaborative Group hospitals and 25 ICUs in 19 of the 29 Tool Kit Group hospitals.

Clinical Tool Use61%49%0.23
BSI Surveillance Guide22/36 (61%)13/25 (52%)0.60
BSI Checklist31/36 (86%)16/25 (64%)0.06
VAP Diagnosis Algorithm24/36 (67%)15/25 (60%)0.60
Ventilator Weaning Protocol23/36 (64%)11/25 (44%)0.18
VAP Surveillance Guide21/36 (58%)12/25 (48%)0.44
VAP Daily Assessment17/36 (47%)6/25 (24%)0.10
Ventilator Weaning Protocol (Flowsheet)15/36 (42%)11/25 (44%)0.99
Data Tools56%30%0.004
QI Implementation Tools19/36 (53%)6/25 (24%)0.03
BSI Statistical Process Control23/36 (64%)5/25 (20%)0.001
VAP Bundle23/36 (64%)11/25 (44%)0.18
VAP Statistical Process Control21/36 (58%)3/25 (12%)0.001
Strategies69%54%0.017
Protocols for BSI24/36 (67%)19/25 (76%)0.57
Protocols for VAP22/36 (61%)9/25 (36%)0.07
Computer Documentation for BSI24/36 (67%)13/25 (52%)0.29
Computer Documentation for VAP25/36 (69%)15/25 (60%)0.58
Increased Staffing3/36 (8%)0/25 (0%)0.26
Written Education for BSI31/36 (86%)19/25 (76%)0.33
Written Education for VAP30/36 (83%)19/25 (76%)0.52
Continuing Education Classes for BSI28/36 (78%)16/25 (64%)0.26
Continuing Education Classes for VAP30/36 (83%)17/25 (68%)0.21
QI teams27/36 (75%)14/25 (56%)0.16
Provider Performance Feedback for BSI23/36 (64%)11/25 (44%)0.18
Provider Performance Feedback for VAP24/36 (67%)11/25 (44%)0.11
Implementation of BSI Checklist28/36 (78%)15/25 (60%)0.16
Implementation of VAP Checklist31/36 (86%)13/25 (52%)0.007

Both groups relied primarily on implementation of protocols and informatics approaches (Table 4) without increasing staff levels. The predominant strategy was education; both groups provided written educational materials and classes to their providers. There was a trend for more Collaborative group members to implement QI teams (Table 4, P = 0.16 compared with the Toolkit group). Although the preponderance of both groups provided feedback reports to their hospital leaders and unit managers, Collaborative group hospitals showed a trend for providing feedback to front‐line providers (P = 0.11). With respect to self‐reported interventions, 78% of the Collaborative ICUs reported implementing a CLABSI checklist and 86% a VAP checklist, whereas only 60% of the Toolkit group reported implementation of a CLABSI checklist (P = 0.16) and 52% a VAP checklist (P = 0.007). Once a tool was implemented, both groups reported a high rate of sustaining the implementation (ranging from 86% to 100%). There also seemed to be a pattern of sequencing the interventions. Initial efforts tend to focus on provider education and evidence‐based protocols. Later efforts include more formal formation of QI teams followed by implementation of checklists. The evidence for sequencing of interventions is qualitative; we lacked subgroup sample size to substantiate these results with statistical analysis.

Discussion

In our investigation of Virtual Collaborative and Toolkit strategies for spreading the implementation of safe practices for CLABSI and VAP, ICUs in the Collaborative group had more complete implementation of the processes for prevention of hospital‐associated infections. Although both groups accessed clinical resources consistent with surveillance and clinical education, the Virtual Collaborative group attended to data and implementation methods more likely to lead to systemic CQI and organizational changes. ICUs that engaged these resources believed them useful in implementing QI, and more than 85% of the practices were sustained once integrated into routine care. Although the Collaborative ICUs were about 50% more likely to implement improvement strategies, these differences in implementation and process of care did not translate into group differences or longitudinal changes in infection rates.

In contrast to the context of our investigation, most published QI studies on health careassociated infection prevention report high baseline rates followed by a significant decline in infection rates.1419 The baseline infection rates in our study hospitals were actually below the endpoint found in many prior studies, suggesting that any marginal effects from our intervention would be more difficult to detect. Our study was implemented during the IHI's 100,000 Lives Campaign,20 a trend that may have brought about these lower baseline rates and thus a tighter margin for improvement.

The median CLABSI baseline rate in the well‐publicized Michigan hospital study was 2.7 per 1000 catheter days.21, 22 Although our baseline rate was similar (2.27 per 1000 catheter days), their reported postintervention rate was near zero, inferring nearly total elimination of the risk for CLABSI within 3‐18 months of study implementation. Several other studies using a collaborative approach have similarly reported high‐performance near‐zero results in reducing VAP23, 24 and CLABSI2528 rates. The difference between the present and previously published near‐zero result outcomes raises questions about collaboration‐based studies. We noticed 2 phenomena. First, there was slow uptake of data‐driven QI, and second, there was a differential uptake between general knowledge (clinical evidence and education) and QI implementation knowledge.29, 30

Lack of infrastructure to support data‐driven QI remains a significant barrier throughout the health care system, and teams in collaboratives often must work intensively toward improving their information systems' capability for the purpose of data‐driven decision support.1, 15, 31, 32 Systematic, standardized collection of CLABSI and VAP outcomes was initially lacking in many of our study hospitals,10 and our project expended early effort to deploy a system‐wide standardized infection control database registry.

Both of our study groups gravitated toward educational training and evidence‐based protocol decision‐support strategies. A focus only on established surveillance and education‐based fixes (eg, asking clinicians to follow a protocol within their existing care processes) have produced 32% to 57% reductions in health careacquired infections.3335 These early gains, however, are unlikely to produce the sustained near‐zero results that some collaborative teams have reported.22, 25

The ability to achieve sustained high‐performance results depends on organizational context and requires time.31 A potential benefit of collaboratives might be the return on investment attained by organizational change in quality and safety climate and its influence across the whole organization.19, 31, 36 Participants requiring systems training in the CQI process may not gain these benefits until well into their collaborative.31 For example, accumulating evidence demonstrates that the use of checklists can reduce errors of omission. Although a checklist seems a simple intervention, its effective implementation into routine care processes actually requires time for system redesign that addresses changes in multidisciplinary roles and responsibilities, frontline clinician and mid‐level management buy‐in, new methods of data collection and feedback, unanticipated involvement of ancillary services (eg, medical records, housekeeping), as well as changes to organizational policies, expectations, and priorities that connect silos of care and integrate hierarchical operations. Wall et al.37 and Pronovost and colleagues19, 21, 22, 25 highlighted the strategic effectiveness of embedding a checklist as a behavioral and data collection tool into frontline care process, leading to a redefined role of nursing, as well as new data for further cycles of improvement that collectively reduced infection rates. In our study, the Virtual Collaborative group did not have greater use of CLABSI and VAP checklists until the QI teams had been formed months into the project, consistent with the hypothesis that beneficial translation of desired changes in process of care to observed improvements in patient outcomes may take longer than 18 months to achieve19, 25, 27, 38 as opposed to the remarkable 3 months reported in the Keystone ICU project.21

Our study has several limitations. Our intervention did not mandate fixed specific components of intervention or QI methods. Each medical center was free to tailor its use of tools and change ideas, producing site variation in implementation methods and investment in support of QI. Like other multicomponent, multidimensional intervention studies, we were not able to test the effectiveness of particular QI components or the thoroughness of surveillance for CLABSI and VAP related to efforts to standardize the approach, and we did not have the resources to monitor the intensity with which participants approached QI. Furthermore, our data were dependent on self‐reports and were not verified by independent assessment of the fidelity with which the interventions were implemented, a checklist was embedded into usual care, or practices were enforced by nurses. In addition, the virtual collaborative circumvents the face‐to‐face learning sessions that might play a role in collaborative social networking, peer pressure, and acculturation.31, 36

Despite these limitations, we found that the Virtual Collaborative performed just like a Breakthrough Collaborative with a gradual uptake of implementation science using QI methods, team management, and statistical process control tools. The Toolkit condition had an even slower uptake. From an organization's perspective, the bottom‐line decision is whether a greater and meaningful proportion of collaborative participants will be successful to justify the investment of effort compared to a toolkit‐only approach. Our findings suggest that organizations engaged in change but lacking expertise in implementation science can potentially benefit from the acculturation, experiential learning, and uptake of QI provided by a collaborative.

In summary, although our Virtual Collaborative intervention was more likely to produce changes in ICU processes of care, there were no improvements in patient outcomes over this 18‐month study. The current popularity of evidence‐based guidelines, care protocols, prevention awareness, and surveillance may have produced a background of secular trend, making it difficult to ascertain effects of our QI intervention. Nonetheless, important lessons can be gleaned from this randomized controlled trial. Our study supports the proposition that as long as organizations vary in their capacity for and commitment to the science of QI and systems engineering, we should anticipate variation, uncertainty, and mixed results from short‐term, rapid cycle initiatives.27, 28, 31, 32, 39, 40 The untested, longer‐term benefit produced by a collaborative may be its stimulation of enduring systems engineering that optimizes an environment for QI of health care processes focused on desired outcomes.

Acknowledgements

The authors thank the Agency for Healthcare Research and Quality collaborative investigators for their work in this study: Xu Lei Liu, MS, at Vanderbilt; Laurie Brewer, RN MBA, Jason Hickok, Steve Horner, Susan Littleton, Patsy McFadden, RN BSN MPA CIC, Steve Mok, PharmD, Jonathan Perlin, MD PhD, Joan Reischel, RN BSN CCRN, and Sheri G. Chernestky Tejedor, MD, and all the HCA medical centers that participated in this project.

References
  1. Shortell SM,Bennett CL,Byck GR.Assessing the impact of continuous quality improvement on clinical practice: What it will take to accelerate progress.Milbank Q.1998;76:593624.
  2. Berwick DM.Continuous improvement as an ideal in health care.N Engl J Med.1989;320:5356.
  3. Kilo CM.A framework for collaborative improvement: Lessons from the Institute for Healthcare Improvement's Breakthrough Series.Qual Manag Health Care.1998;6(4):113.
  4. Ayers LR,Beyea SC,Godfrey MM,Harper DC,Nelson EC,Batalden PB.Quality improvement learning collaboratives.Qual Manag Health Care.2005;14:234237.
  5. Boushon B,Provost L,Gagnon J,Carver P.Using a virtual breakthrough series collaborative to improve access in primary care.Jt Comm J Qual Patient Saf.2006;32:573584.
  6. Eagle KA,Gallogly M,Mehta RH, et al.Taking the national guideline for care of acute myocardial infarction to the bedside: Developing the guideline applied in practice (GAP) initiative in Southeast Michigan.Jt Comm J Qual Improv.2002;28:519.
  7. Adams K,Corrigan JM.Priority Areas for National Action: Transforming Health Care Quality.Washington, DC:The National Academies Press;2003.
  8. Greevy RA,Lu B,Silber SH,Rosenbaum P.Optimal multivariate matching before randomization.Biostatistics.2004;5:263275.
  9. Institute for Healthcare Improvement. The 100,000 lives campaign. http://www.ihi.org/IHI/Programs/Campaign.htm;2005.
  10. Talbot T,Tejedor SC,Greevy RA, et al.Survey of infection control programs in a large, national healthcare system.Infect Control Hosp Epidemiol.2007;28:14011403.
  11. Shojania KG,McDonald KM,Wachter RM,Owens DK. Closing the quality gap: A critical analysis of quality improvement strategies, Volume 1‐Series overview and methodology. Technical Review 9 (Contract No 290–02‐0017 to the Stanford University‐UCSF Evidence‐based Practices Center), 2004. www.ahrq.gov/clinic/tp/qgap1tp.htm. Accessed November 11,2010.
  12. Mohr JJ,Batalden PB.Improving safety on the front lines: the role of clinical microsystems.Qual Saf Health Care.2002;11:4550.
  13. Borenstein M,Rothstein H,Cohen J.SamplePower 2.0.Chicago, IL:SPSS Inc.;2001.
  14. Berriel‐Cass D,Adkins FW,Jones P,Fakih MG.Eliminating nosocomial infections at Ascension Health.Jt Comm J Qual Patient Saf.2006;32:612620.
  15. Bonello RS,Fletcher CE,Becker WK, et al.An intensive care unit quality improvement collaborative in nine department of Veterans Affairs hospitals: reducing ventilator‐associated pneumonia and catheter‐related bloodstream infection rates.Jt Comm J Qual Patient Saf.2008;34:639645.
  16. Cocanour CS,Peninger M,Domonoske BD, et al.Decreasing ventilator‐associated pneumonia in a trauma ICU.J Trauma.2006;61:122130.
  17. Frankel HL,Crede WB,Topal JE,Roumanis SA,Devlin MW,Foley AB.Use of corporate six sigma performance‐improvement strategies to reduce incidence of catheter‐related bloodstream infections in a surgical ICU.J Am Coll Surg.2005;201:349358.
  18. Jain M,Miller L,Belt D,King D,Berwick DM.Decline in ICU adverse events, nosocomial infections and cost through a quality improvement initiative focusing on teamwork and culture change.Qual Saf Health Care.2006;15:235239.
  19. Resar R,Pronovost PJ,Harden C,Simmonds R,Rainey T,Nolan TW.Using a bundle approach to improve ventilator care processes and reduce ventilator‐associated pneumonia.Jt Comm J Qual Patient Saf.2005;31:243248.
  20. Berwick DM,Calkins DR,McCannon CJ,Hackbarth AD.The 100000 lives campaign: Setting a goal and a deadline for improving health care quality.JAMA.2006;295:324327.
  21. Pronovost PJ,Needham D,Berenholtz SM.An intervention to decrease catheter‐related bloodstream infections in the ICU.N Engl J Med.2006;355:27252732.
  22. Pronovost PJ,Berenholtz SM,Goeschel C, et al.Improving patient safety units in Michigan.J Crit Care.2008;23:207221.
  23. Fox MY.Toward a zero VAP rate: Personal and team approaches in the ICU.Crit Care Nurs Q.2006;29:108114.
  24. Youngquist P,Carroll M,Farber M, et al.Implementing a ventilator bundle in a community hospital.Jt Comm J Qual Patient Saf.2007;33:219225.
  25. Berenholtz SM,Pronovost PJ,Lipsett PA, et al.Eliminating catheter‐realted bloodstream infections in the intensive care unit.Crit Care Med.2004;32:20142020.
  26. Harnage S.Innovative bundle wipes out catheter‐related bloodstream infections.Nursing.2008;38:1718.
  27. Koll BS,Straub TA,Jalon HS,Block R,Heller KS,Ruiz RE.The CLABs collaborative: a regionwide effort to improve the quality of care in hospitals.Jt Comm J Qual Patient Saf.2008;34:713723.
  28. Render ML,Brungs S,Kotagal U, et al.Evidence‐based practice to reduce central line infections.Jt Comm J Qual Patient Saf.2006;32:253260.
  29. Nembhard IM.Learning and improving in quality improvement collaboratives: which collaborative features do participants value most?Health Serv Res.2009;44(2 Pt 1):359378.
  30. Grossman E,Keegan T,Lesser AL, et al.Inside the health disparities collaboratives: a detailed exploration of quality improvement at community health centers.Med Care.2008;46:489496.
  31. Ovretveit J,Bate P,Cleary P, et al.Quality collaboratives: lessons from research.Qual Saf Health Care.2002;11:345351.
  32. Pearson ML,Wu S,Schaefer CT, et al.Assessing the implementation of the chronic care model in quality improvement collaboratives.Health Serv Res.2005;40:978996.
  33. Gastmeier P,Geffers C.Prevention of ventilator‐associated pneumonia: Analysis of studies published since 2004.J Hosp Infect.2007;67:18.
  34. McKinley LL,Moriarty HJ,Short TH,Johnson CC.Effect of comparative data feedback on intensive care unit infection rates in a Veterans Administration Hospital network system.Am J Infect Control.2003;31:397404.
  35. Salahuddin N,Zafar A,Sukhyani L, et al.Reducing ventilator‐associated pneumonia rates through a staff education programme.J Hosp Infect.2004;57:223227.
  36. Alexander JA,Weiner BJ,Shortell SM,Baker LC.Does quality improvement implementation affect hospital quality of care?Hosp Top.2007;85:312.
  37. Wall RJ,Ely EW,Ellis D,Dittus RS,Foss J,Speroff T.Using real‐time process measurements to reduce catheter‐related bloodstream infections in the intensive care unit.Qual Saf Health Care.2005;14:295302.
  38. Esmail R,Duchscherer G,Giesbrecht J,King J,Ritchie P,Zuege D.Prevention of ventilator‐associated pneumonia in the Calgary health region: a Canadian success story!Healthcare Qual.2008;11(3 Spec No):129136.
  39. Mittman BS.Creating the evidence base for quality improvement collaboratives.Ann Intern Med.2004;140:897901.
  40. Schouten LMT,Hulscher MEJL,Everdigen JJE,Huijsman R,Grol RPTM.Evidence for the impact of quality improvement collaboratives: systematic review.Br Med J.2008;336:14911494.
Article PDF
Issue
Journal of Hospital Medicine - 6(5)
Publications
Page Number
271-278
Legacy Keywords
patient safety, quality improvement, central line–associated bloodstream infection, ventilator‐associated pneumonia
Sections
Article PDF
Article PDF

Continuous quality improvement (CQI) methodologies provide a framework for initiating and sustaining improvements in complex systems.1 By definition, CQI engages frontline staff in iterative problem solving using plandostudyact cycles of learning, with decision‐making based on real‐time process measurements.2 The Institute for Healthcare Improvement (IHI) has sponsored Breakthrough Series Collaboratives since 1996 to accelerate the uptake and impact of quality improvement (QI).3, 4 These collaboratives are typically guided by evidence‐based clinical practice guidelines, incorporate change methodologies, and rely on clinical and process improvement subject matter experts. Through the collaborative network, teams share knowledge and ideas about effective and ineffective interventions as well as strategies for overcoming barriers. The collaborative curriculum includes CQI methodology, multidisciplinary teamwork, leadership support, and tools for measurement. Participants are typically required to invest resources and send teams to face‐to‐face goal‐oriented meetings. It is costly for a large healthcare organization to incorporate travel to a learning session conference into its collaborative model. Thus, we attempted virtual learning sessions that make use of webcasts, a Web site, and teleconference calls for tools and networking.5

A recent derivative of collaboratives has been deployment of toolkits for QI. Intuition suggests that such toolkits may help to enable change, and thus some agencies advocate the simpler approach of disseminating toolkits as a change strategy.6 Toolkit dissemination is a passive approach in contrast to collaborative participation, and its effectiveness has not been critically examined in evidence‐based literature.

We sought to compare the virtual collaborative model with the toolkit model for improving care. Recommendations and guidelines for central lineassociated bloodstream infection (CLABSI) and ventilator‐associated pneumonia (VAP) prevention have not been implemented reliably, resulting in unnecessary intensive care unit (ICU) morbidity and mortality and fostering a national call for improvement.7 Our aim was to compare the effectiveness of the virtual collaborative and toolkit approaches on preventing CLABSI and VAP in the ICU.

Methods

This cluster randomized trial included medical centers within the Hospital Corporation of America (HCA), a network of hospitals located primarily in the southern United States. To minimize contamination bias between study groups within the same facility, the unit of randomization was the hospital and implementation was at the level of the ICU. The project received approval from the Vanderbilt University Institutional Review Board.

Leaders of all medical centers with at least 1 adult or pediatric ICU received an invitation from HCA leadership to participate in a QI initiative. Facility clinicians and managers completed baseline surveys (shown in the Supporting Information) on hospital characteristics, types of ICUs, patient safety climate, and QI resources between July and November 2005. Hospital‐level data were extracted from the enterprise‐wide data warehouse. Hospitals willing to participate were matched on geographic location and ICU volume and then randomized into either the Virtual Collaborative (n = 31) or Toolkit (n = 30) groups in December 20058; 1 of the hospitals was sold, yielding 29 hospitals in the Toolkit (n = 29) group. The study lasted 18 months from January 2006 through September 2007, with health careassociated infection data collected through December 2007, and follow‐up data collection through April 2008.

The QI initiative included educational opportunities, evidence‐based clinical prevention interventions, and processes and tools to implement and measure the impact of these interventions. Participants in both groups were offered interactive Web seminars during the study period; 5 of these seminars were on clinical subject matter, and 5 seminars were on patient safety, charting use of statistical process control and QI methods. The interventions were evidence‐based care bundles.9 The key interventions for preventing CLABSI were routine hand hygiene, use of chlorhexidine skin antisepsis, maximal barrier precautions during catheter insertion, catheter site and care, and avoidance of routine replacement of catheters. The key interventions to prevent VAP were routine elevation of head of the bed, regular oral care, daily sedation vacations, daily assessment of readiness to extubate, secretion cleaning, peptic ulcer disease prophylaxis, and deep vein thrombosis prophylaxis.

Toolkit Group

Hospitals randomized to this arm received a toolkit during study month 1 containing a set of evidence‐based guidelines and fact sheets for preventing CLABSI and VAP, a review of QI and teamwork methods, standardized data collection tools, and standardized charting tools. The nurse and quality managers for the Toolkit ICUs were provided ad libitum access to the HCA intranet toolkit Web site containing all of the educational seminars, clinical tools, and QI tools. Otherwise, ICUs in this group were on their own to initiate and implement a local hospital QI initiative to prevent CLABSI and VAP.

Virtual Collaborative Group

In addition to the materials and Web site support described above, facility leaders and managers in this Virtual Collaborative group agreed to participate in a virtual collaborative to develop processes to more reliably implement evidence‐based interventions to prevent CLABSI and VAP. The collaboration differed from the Breakthrough Series model3, 4 in that teams did not come together for face‐to‐face educational and planning sessions but instead attended Web seminars and teleconferences for reporting back to the larger group.5 Teams were supported through monthly educational and troubleshooting conference calls, individual coaching coordinated by the HCA corporate office of quality, safety, and performance improvement, and an e‐mail listserv designed to stimulate interaction among teams.

Clinical Outcome Measures

Although most participating hospitals defined CLABSI and VAP using the Centers for Disease Control and Prevention definitions, data collection and surveillance methods varied across hospitals.10 Education was provided to standardize outcome measurement. A data registry Web application was created as a new tool for infection control data entry, and healthcare‐associated infection data reporting by the infection control personnel was mandated starting the first quarter of 2006. To verify electronic data and correct missing information, the infection control personnel were requested to complete a retrospective data collection sheet providing quarterly reports from January 2005 through December 2007 on ICU infection events as well as total catheter days and ventilator days to allow calculation of event rates. Outcome measures of CLABSI and VAP were at the level of the hospital.

Follow‐Up

The HCA e‐mail distribution and collection routine was employed for the follow‐up survey of ICU nurse and quality managers for all participating medical centers from January 2008 through April 2008. A single survey (shown in the Supporting Information) was requested from each participating ICU. The ICU‐level surveys included questions about the implementation of the CLABSI and VAP process interventions, access of tools, participation in Web seminars, and use of QI strategies.11, 12 The postintervention survey also assessed the character and amount of implementation and teamwork activity expended.

Median CLABSI and VAP rates for a 3‐month baseline and quarterly postintervention periods were compared between the 2 study groups. The CLABSI and VAP infection rates were also analyzed using hierarchical negative binomial regression models to model infection rate changes over time (time in months and group by time interaction effects) and account for clustering of ICUs within hospitals and adjusting for baseline covariates. Baseline and process variables at the hospital and ICU level were compared using chi‐square tests and t tests according to the type of measurement. Time‐to‐event analyses were conducted to compare the groups on time to initiation of a care process. All analyses were conducted using the (R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2010).

The power of the study was calculated a priori with a 1‐tailed alpha of 0.05 and group size of 30. We hypothesized a 50% decrease in hospital‐associated infection rates for the Collaborative group vs. a 10% to 15% decrease for the Toolkit group. The calculations yielded power ranging from a low of 82% to a high of 91% for testing group differences.13

Results

Participating facilities included rural (11%), inner city (28%), and suburban (61%) medical centers. The 60 participating sites did not differ in administrative variables from the 113 nonparticipating HCA sites (results not shown). The median hospital size was 177 beds and the median ICU size was 16 beds. The hospitals did not differ between study groups (Table 1). At baseline, 45% of the facilities reported having a CLABSI program and 62% a VAP program.

Baseline Characteristics of the Virtual Collaborative and Toolkit Groups
Hospital Factors at BaselineVirtual CollaborativeToolkitP Value
  • Abbreviations: IQR, interquartile range; SD, standard deviation.

  • One of the 30 hospitals randomized to the Toolkit group was subsequently sold, resulting in 29 hospitals for this study condition.

Number of hospitals3129* 
ICU annual patient volume, median (IQR)568 (294, 904)578 (244, 1077)0.93
ICU patient length of stay days, median (IQR)3882 (1758, 5718)4228 (1645, 6725)0.95
ICU mortality rate, percent (SD)5.7% (3.1%)7.1% (3.6%)0.13
Medicare/Medicaid, percent (SD)68.6% (9.5%)68.5% (10.1%)0.95
Percent admitted to ICU from the Emergency Department (SD)71% (15%)67% (20%)0.27
Percent female (SD)49.7% (5.7%)50.3% (7.7%)0.79
Medicare case‐mix weight, mean (SD)1221 (1007)1295 (1110)0.82
Percent hospitalist ICU management47%40%0.61

The baseline and quarterly median and pooled infection rates for the Toolkit and Collaboration groups are shown in Table 2 for CLABSI and in Table 3 for VAP. There were no significant differences in the baseline rates for either CLABSI (P = 0.24) or VAP (P = 0.72) between the Collaborative and Toolkit groups. There was no significant change for either CLABSI or VAP outcomes at either 12 or 18 months of follow‐up. The median bloodstream infection rate for all participating hospitals was 2.27 at baseline, 1.18 at 12 months (P = 0.13), and 2.23 per 1000 catheter days 18 months later (P = 0.95). The median VAP rate for participating hospitals was 2.90 at baseline, 2.67 at 12 months (P = 0.44), and 2.52 per 1000 ventilator days 18 months later (P = 0.84). The hierarchical regression analysis found that neither the Collaborative nor Toolkit groups improved CLABSI (P = 0.75 and P = 0.83, respectively) or VAP (P = 0.61 and P = 0.37, respectively) rates over time, and there was no differential performance between the 2 groups for either outcome (bloodstream infection, P = 0.71; VAP, P = 0.80).

CLABSI Rates, per 1000 Catheter Days, Overall and by Study Group
 OverallVirtual CollaborativeToolkit
 N = 59 HospitalsN = 30 HospitalsN = 29 Hospitals
Study PeriodHospital Median (IQR)Rate Pooled Across HospitalsHospital Median (IQR)Rate Pooled Across HospitalsHospital Median (IQR)Rate Pooled Across Hospitals
  • Abbreviation: IQR, interquartile range.

Baseline2.27 (0.00‐3.98)2.421.84 (0.00‐3.83)1.672.42 (0.65‐6.80)3.05
3 Month2.27 (1.30‐4.69)2.612.24 (0.54‐4.69)2.342.47 (1.48‐5.35)2.85
6 Month2.37 (0.00‐4.29)2.732.28 (0.00‐3.73)2.352.54 (0.00‐4.98)3.09
9 Month1.66 (0.00‐3.84)2.451.76 (0.00‐3.74)2.281.23 (0.00‐3.93)2.59
12 Month1.18 (0.00‐3.10)2.171.18 (0.00‐2.71)1.721.17 (0.00‐3.61)2.58
15 Month1.93 (0.00‐4.25)2.292.04 (0.00‐4.91)2.531.77 (0.00‐3.30)2.08
18 Month2.23 (0.00‐4.97)2.732.76 (0.00‐4.67)2.751.16 (0.00‐5.46)2.72
VAP Rates per 1000 Ventilator Days, Overall and by Study Group
Study PeriodOverallVirtual CollaborativeToolkit
N = 59 HospitalsN = 30 HospitalsN = 29 Hospitals
Hospital Median (IQR)Rate Pooled Across HospitalsHospital Median (IQR)Rate Pooled Across HospitalsHospital Median (IQR)Rate Pooled Across Hospitals
  • Abbreviation: IQR, interquartile range.

Baseline2.90 (0.00‐6.14)3.972.14 (0.00‐6.09)3.433.49 (0.00‐7.04)4.36
3 Month3.12 (0.00‐8.40)4.463.01 (0.00‐9.11)4.223.32 (0.00‐8.25)4.62
6 Month3.40 (0.00‐7.53)4.972.72 (0.00‐7.09)4.814.61 (0.00‐9.37)5.10
9 Month1.49 (0.00‐4.87)2.990 (0.00‐3.94)2.512.27 (0.00‐6.27)3.36
12 Month2.67 (0.00‐4.60)4.392.67 (0.00‐4.47)3.822.66 (0.00‐4.82)4.95
15 Month3.06 (0.00‐5.10)4.032.40 (0.00‐3.94)3.573.65 (1.15‐6.57)4.45
18 Month2.52 (0.00‐7.45)4.612.93 (0.00‐7.63)5.022.06 (0.00‐6.59)4.31

The poststudy survey was completed by 27 of 31 (87%) of Collaborative group hospitals and 19 of the 29 (66%) Toolkit hospitals. Both groups reported QI improvement efforts to prevent CLABSI (Collaborative 97% vs. Toolkit 88%, P = 0.29) and VAP (Collaborative 97% vs. Toolkit 96%, P = 0.99). Eighty‐three percent of the Collaborative group implemented all components of the bloodstream infection prevention interventions compared with 64% for the Toolkit group (P = 0.13; Figure 1). The Collaborative group implemented daily catheter review more often than the Toolkit group (P = 0.04) and began the process implementation sooner following study implementation (P = 0.006 vs. Toolkit; see Supporting Information Figure). Eighty‐six percent of the Collaborative group implemented the complete VAP prevention interventions vs. 64% of the Toolkit group (P = 0.06; Figure 1) and the Collaborative group conducted the sedation vacation intervention more often (P = 0.03).

Figure 1
(A) Follow‐up survey of self‐reported implementation of key CLABSI prevention interventions by study group. (B) Follow‐up survey of self‐reported implementation of key VAP prevention interventions by study group.

The Collaborative group participated in 57% of the seminars, whereas the Toolkit group participated in 39% (P = 0.014). Members of both groups attended more than half the clinical topics (Collaborative 64% vs. Toolkit 56%, P = 0.37). The Collaborative group had greater participation in the data and method topics (Collaborative 50% vs. Toolkit 22%, P < 0.001). The proportion of hospitals finding the seminars useful to their QI efforts was 49% for the Collaborative and 30% for the Toolkit group (P = 0.017). When restricted to hospitals that participated in the seminars, the usefulness rating was higher for both clinical (91% for the Collaborative and 86% for Toolkit) and Data/Methods (79% for Collaborative and 55% for Toolkit) topics.

A set of 14 tools were produced during the study period (Table 4); 9 clinically related tools (eg, checklists, algorithms, protocols, and flowsheets) and 5 data monitoring and quality improvement tools (eg, easy‐to‐use statistical process control spreadsheet templates, quality improvement tools, and computer tools). The Collaborative group downloaded a median of 10 tools and the Toolkit group a median of 7 (P = 0.051). The groups did not differ in their access to the clinical tools (P = 0.23) but the Collaborative group accessed a greater proportion of the data/methods tools (P = 0.004).

Follow‐up Survey on Study Groups' Tool Use and Strategies for Improvement
Tool Access and StrategiesCollaborative HospitalsaTool Kit HospitalsaP‐value
N = 36 ICUsN = 25 ICUs
  • Post‐survey respondents included 36 ICUs in 26 of the 30 Collaborative Group hospitals and 25 ICUs in 19 of the 29 Tool Kit Group hospitals.

Clinical Tool Use61%49%0.23
BSI Surveillance Guide22/36 (61%)13/25 (52%)0.60
BSI Checklist31/36 (86%)16/25 (64%)0.06
VAP Diagnosis Algorithm24/36 (67%)15/25 (60%)0.60
Ventilator Weaning Protocol23/36 (64%)11/25 (44%)0.18
VAP Surveillance Guide21/36 (58%)12/25 (48%)0.44
VAP Daily Assessment17/36 (47%)6/25 (24%)0.10
Ventilator Weaning Protocol (Flowsheet)15/36 (42%)11/25 (44%)0.99
Data Tools56%30%0.004
QI Implementation Tools19/36 (53%)6/25 (24%)0.03
BSI Statistical Process Control23/36 (64%)5/25 (20%)0.001
VAP Bundle23/36 (64%)11/25 (44%)0.18
VAP Statistical Process Control21/36 (58%)3/25 (12%)0.001
Strategies69%54%0.017
Protocols for BSI24/36 (67%)19/25 (76%)0.57
Protocols for VAP22/36 (61%)9/25 (36%)0.07
Computer Documentation for BSI24/36 (67%)13/25 (52%)0.29
Computer Documentation for VAP25/36 (69%)15/25 (60%)0.58
Increased Staffing3/36 (8%)0/25 (0%)0.26
Written Education for BSI31/36 (86%)19/25 (76%)0.33
Written Education for VAP30/36 (83%)19/25 (76%)0.52
Continuing Education Classes for BSI28/36 (78%)16/25 (64%)0.26
Continuing Education Classes for VAP30/36 (83%)17/25 (68%)0.21
QI teams27/36 (75%)14/25 (56%)0.16
Provider Performance Feedback for BSI23/36 (64%)11/25 (44%)0.18
Provider Performance Feedback for VAP24/36 (67%)11/25 (44%)0.11
Implementation of BSI Checklist28/36 (78%)15/25 (60%)0.16
Implementation of VAP Checklist31/36 (86%)13/25 (52%)0.007

Both groups relied primarily on implementation of protocols and informatics approaches (Table 4) without increasing staff levels. The predominant strategy was education; both groups provided written educational materials and classes to their providers. There was a trend for more Collaborative group members to implement QI teams (Table 4, P = 0.16 compared with the Toolkit group). Although the preponderance of both groups provided feedback reports to their hospital leaders and unit managers, Collaborative group hospitals showed a trend for providing feedback to front‐line providers (P = 0.11). With respect to self‐reported interventions, 78% of the Collaborative ICUs reported implementing a CLABSI checklist and 86% a VAP checklist, whereas only 60% of the Toolkit group reported implementation of a CLABSI checklist (P = 0.16) and 52% a VAP checklist (P = 0.007). Once a tool was implemented, both groups reported a high rate of sustaining the implementation (ranging from 86% to 100%). There also seemed to be a pattern of sequencing the interventions. Initial efforts tend to focus on provider education and evidence‐based protocols. Later efforts include more formal formation of QI teams followed by implementation of checklists. The evidence for sequencing of interventions is qualitative; we lacked subgroup sample size to substantiate these results with statistical analysis.

Discussion

In our investigation of Virtual Collaborative and Toolkit strategies for spreading the implementation of safe practices for CLABSI and VAP, ICUs in the Collaborative group had more complete implementation of the processes for prevention of hospital‐associated infections. Although both groups accessed clinical resources consistent with surveillance and clinical education, the Virtual Collaborative group attended to data and implementation methods more likely to lead to systemic CQI and organizational changes. ICUs that engaged these resources believed them useful in implementing QI, and more than 85% of the practices were sustained once integrated into routine care. Although the Collaborative ICUs were about 50% more likely to implement improvement strategies, these differences in implementation and process of care did not translate into group differences or longitudinal changes in infection rates.

In contrast to the context of our investigation, most published QI studies on health careassociated infection prevention report high baseline rates followed by a significant decline in infection rates.1419 The baseline infection rates in our study hospitals were actually below the endpoint found in many prior studies, suggesting that any marginal effects from our intervention would be more difficult to detect. Our study was implemented during the IHI's 100,000 Lives Campaign,20 a trend that may have brought about these lower baseline rates and thus a tighter margin for improvement.

The median CLABSI baseline rate in the well‐publicized Michigan hospital study was 2.7 per 1000 catheter days.21, 22 Although our baseline rate was similar (2.27 per 1000 catheter days), their reported postintervention rate was near zero, inferring nearly total elimination of the risk for CLABSI within 3‐18 months of study implementation. Several other studies using a collaborative approach have similarly reported high‐performance near‐zero results in reducing VAP23, 24 and CLABSI2528 rates. The difference between the present and previously published near‐zero result outcomes raises questions about collaboration‐based studies. We noticed 2 phenomena. First, there was slow uptake of data‐driven QI, and second, there was a differential uptake between general knowledge (clinical evidence and education) and QI implementation knowledge.29, 30

Lack of infrastructure to support data‐driven QI remains a significant barrier throughout the health care system, and teams in collaboratives often must work intensively toward improving their information systems' capability for the purpose of data‐driven decision support.1, 15, 31, 32 Systematic, standardized collection of CLABSI and VAP outcomes was initially lacking in many of our study hospitals,10 and our project expended early effort to deploy a system‐wide standardized infection control database registry.

Both of our study groups gravitated toward educational training and evidence‐based protocol decision‐support strategies. A focus only on established surveillance and education‐based fixes (eg, asking clinicians to follow a protocol within their existing care processes) have produced 32% to 57% reductions in health careacquired infections.3335 These early gains, however, are unlikely to produce the sustained near‐zero results that some collaborative teams have reported.22, 25

The ability to achieve sustained high‐performance results depends on organizational context and requires time.31 A potential benefit of collaboratives might be the return on investment attained by organizational change in quality and safety climate and its influence across the whole organization.19, 31, 36 Participants requiring systems training in the CQI process may not gain these benefits until well into their collaborative.31 For example, accumulating evidence demonstrates that the use of checklists can reduce errors of omission. Although a checklist seems a simple intervention, its effective implementation into routine care processes actually requires time for system redesign that addresses changes in multidisciplinary roles and responsibilities, frontline clinician and mid‐level management buy‐in, new methods of data collection and feedback, unanticipated involvement of ancillary services (eg, medical records, housekeeping), as well as changes to organizational policies, expectations, and priorities that connect silos of care and integrate hierarchical operations. Wall et al.37 and Pronovost and colleagues19, 21, 22, 25 highlighted the strategic effectiveness of embedding a checklist as a behavioral and data collection tool into frontline care process, leading to a redefined role of nursing, as well as new data for further cycles of improvement that collectively reduced infection rates. In our study, the Virtual Collaborative group did not have greater use of CLABSI and VAP checklists until the QI teams had been formed months into the project, consistent with the hypothesis that beneficial translation of desired changes in process of care to observed improvements in patient outcomes may take longer than 18 months to achieve19, 25, 27, 38 as opposed to the remarkable 3 months reported in the Keystone ICU project.21

Our study has several limitations. Our intervention did not mandate fixed specific components of intervention or QI methods. Each medical center was free to tailor its use of tools and change ideas, producing site variation in implementation methods and investment in support of QI. Like other multicomponent, multidimensional intervention studies, we were not able to test the effectiveness of particular QI components or the thoroughness of surveillance for CLABSI and VAP related to efforts to standardize the approach, and we did not have the resources to monitor the intensity with which participants approached QI. Furthermore, our data were dependent on self‐reports and were not verified by independent assessment of the fidelity with which the interventions were implemented, a checklist was embedded into usual care, or practices were enforced by nurses. In addition, the virtual collaborative circumvents the face‐to‐face learning sessions that might play a role in collaborative social networking, peer pressure, and acculturation.31, 36

Despite these limitations, we found that the Virtual Collaborative performed just like a Breakthrough Collaborative with a gradual uptake of implementation science using QI methods, team management, and statistical process control tools. The Toolkit condition had an even slower uptake. From an organization's perspective, the bottom‐line decision is whether a greater and meaningful proportion of collaborative participants will be successful to justify the investment of effort compared to a toolkit‐only approach. Our findings suggest that organizations engaged in change but lacking expertise in implementation science can potentially benefit from the acculturation, experiential learning, and uptake of QI provided by a collaborative.

In summary, although our Virtual Collaborative intervention was more likely to produce changes in ICU processes of care, there were no improvements in patient outcomes over this 18‐month study. The current popularity of evidence‐based guidelines, care protocols, prevention awareness, and surveillance may have produced a background of secular trend, making it difficult to ascertain effects of our QI intervention. Nonetheless, important lessons can be gleaned from this randomized controlled trial. Our study supports the proposition that as long as organizations vary in their capacity for and commitment to the science of QI and systems engineering, we should anticipate variation, uncertainty, and mixed results from short‐term, rapid cycle initiatives.27, 28, 31, 32, 39, 40 The untested, longer‐term benefit produced by a collaborative may be its stimulation of enduring systems engineering that optimizes an environment for QI of health care processes focused on desired outcomes.

Acknowledgements

The authors thank the Agency for Healthcare Research and Quality collaborative investigators for their work in this study: Xu Lei Liu, MS, at Vanderbilt; Laurie Brewer, RN MBA, Jason Hickok, Steve Horner, Susan Littleton, Patsy McFadden, RN BSN MPA CIC, Steve Mok, PharmD, Jonathan Perlin, MD PhD, Joan Reischel, RN BSN CCRN, and Sheri G. Chernestky Tejedor, MD, and all the HCA medical centers that participated in this project.

Continuous quality improvement (CQI) methodologies provide a framework for initiating and sustaining improvements in complex systems.1 By definition, CQI engages frontline staff in iterative problem solving using plandostudyact cycles of learning, with decision‐making based on real‐time process measurements.2 The Institute for Healthcare Improvement (IHI) has sponsored Breakthrough Series Collaboratives since 1996 to accelerate the uptake and impact of quality improvement (QI).3, 4 These collaboratives are typically guided by evidence‐based clinical practice guidelines, incorporate change methodologies, and rely on clinical and process improvement subject matter experts. Through the collaborative network, teams share knowledge and ideas about effective and ineffective interventions as well as strategies for overcoming barriers. The collaborative curriculum includes CQI methodology, multidisciplinary teamwork, leadership support, and tools for measurement. Participants are typically required to invest resources and send teams to face‐to‐face goal‐oriented meetings. It is costly for a large healthcare organization to incorporate travel to a learning session conference into its collaborative model. Thus, we attempted virtual learning sessions that make use of webcasts, a Web site, and teleconference calls for tools and networking.5

A recent derivative of collaboratives has been deployment of toolkits for QI. Intuition suggests that such toolkits may help to enable change, and thus some agencies advocate the simpler approach of disseminating toolkits as a change strategy.6 Toolkit dissemination is a passive approach in contrast to collaborative participation, and its effectiveness has not been critically examined in evidence‐based literature.

We sought to compare the virtual collaborative model with the toolkit model for improving care. Recommendations and guidelines for central lineassociated bloodstream infection (CLABSI) and ventilator‐associated pneumonia (VAP) prevention have not been implemented reliably, resulting in unnecessary intensive care unit (ICU) morbidity and mortality and fostering a national call for improvement.7 Our aim was to compare the effectiveness of the virtual collaborative and toolkit approaches on preventing CLABSI and VAP in the ICU.

Methods

This cluster randomized trial included medical centers within the Hospital Corporation of America (HCA), a network of hospitals located primarily in the southern United States. To minimize contamination bias between study groups within the same facility, the unit of randomization was the hospital and implementation was at the level of the ICU. The project received approval from the Vanderbilt University Institutional Review Board.

Leaders of all medical centers with at least 1 adult or pediatric ICU received an invitation from HCA leadership to participate in a QI initiative. Facility clinicians and managers completed baseline surveys (shown in the Supporting Information) on hospital characteristics, types of ICUs, patient safety climate, and QI resources between July and November 2005. Hospital‐level data were extracted from the enterprise‐wide data warehouse. Hospitals willing to participate were matched on geographic location and ICU volume and then randomized into either the Virtual Collaborative (n = 31) or Toolkit (n = 30) groups in December 20058; 1 of the hospitals was sold, yielding 29 hospitals in the Toolkit (n = 29) group. The study lasted 18 months from January 2006 through September 2007, with health careassociated infection data collected through December 2007, and follow‐up data collection through April 2008.

The QI initiative included educational opportunities, evidence‐based clinical prevention interventions, and processes and tools to implement and measure the impact of these interventions. Participants in both groups were offered interactive Web seminars during the study period; 5 of these seminars were on clinical subject matter, and 5 seminars were on patient safety, charting use of statistical process control and QI methods. The interventions were evidence‐based care bundles.9 The key interventions for preventing CLABSI were routine hand hygiene, use of chlorhexidine skin antisepsis, maximal barrier precautions during catheter insertion, catheter site and care, and avoidance of routine replacement of catheters. The key interventions to prevent VAP were routine elevation of head of the bed, regular oral care, daily sedation vacations, daily assessment of readiness to extubate, secretion cleaning, peptic ulcer disease prophylaxis, and deep vein thrombosis prophylaxis.

Toolkit Group

Hospitals randomized to this arm received a toolkit during study month 1 containing a set of evidence‐based guidelines and fact sheets for preventing CLABSI and VAP, a review of QI and teamwork methods, standardized data collection tools, and standardized charting tools. The nurse and quality managers for the Toolkit ICUs were provided ad libitum access to the HCA intranet toolkit Web site containing all of the educational seminars, clinical tools, and QI tools. Otherwise, ICUs in this group were on their own to initiate and implement a local hospital QI initiative to prevent CLABSI and VAP.

Virtual Collaborative Group

In addition to the materials and Web site support described above, facility leaders and managers in this Virtual Collaborative group agreed to participate in a virtual collaborative to develop processes to more reliably implement evidence‐based interventions to prevent CLABSI and VAP. The collaboration differed from the Breakthrough Series model3, 4 in that teams did not come together for face‐to‐face educational and planning sessions but instead attended Web seminars and teleconferences for reporting back to the larger group.5 Teams were supported through monthly educational and troubleshooting conference calls, individual coaching coordinated by the HCA corporate office of quality, safety, and performance improvement, and an e‐mail listserv designed to stimulate interaction among teams.

Clinical Outcome Measures

Although most participating hospitals defined CLABSI and VAP using the Centers for Disease Control and Prevention definitions, data collection and surveillance methods varied across hospitals.10 Education was provided to standardize outcome measurement. A data registry Web application was created as a new tool for infection control data entry, and healthcare‐associated infection data reporting by the infection control personnel was mandated starting the first quarter of 2006. To verify electronic data and correct missing information, the infection control personnel were requested to complete a retrospective data collection sheet providing quarterly reports from January 2005 through December 2007 on ICU infection events as well as total catheter days and ventilator days to allow calculation of event rates. Outcome measures of CLABSI and VAP were at the level of the hospital.

Follow‐Up

The HCA e‐mail distribution and collection routine was employed for the follow‐up survey of ICU nurse and quality managers for all participating medical centers from January 2008 through April 2008. A single survey (shown in the Supporting Information) was requested from each participating ICU. The ICU‐level surveys included questions about the implementation of the CLABSI and VAP process interventions, access of tools, participation in Web seminars, and use of QI strategies.11, 12 The postintervention survey also assessed the character and amount of implementation and teamwork activity expended.

Median CLABSI and VAP rates for a 3‐month baseline and quarterly postintervention periods were compared between the 2 study groups. The CLABSI and VAP infection rates were also analyzed using hierarchical negative binomial regression models to model infection rate changes over time (time in months and group by time interaction effects) and account for clustering of ICUs within hospitals and adjusting for baseline covariates. Baseline and process variables at the hospital and ICU level were compared using chi‐square tests and t tests according to the type of measurement. Time‐to‐event analyses were conducted to compare the groups on time to initiation of a care process. All analyses were conducted using the (R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2010).

The power of the study was calculated a priori with a 1‐tailed alpha of 0.05 and group size of 30. We hypothesized a 50% decrease in hospital‐associated infection rates for the Collaborative group vs. a 10% to 15% decrease for the Toolkit group. The calculations yielded power ranging from a low of 82% to a high of 91% for testing group differences.13

Results

Participating facilities included rural (11%), inner city (28%), and suburban (61%) medical centers. The 60 participating sites did not differ in administrative variables from the 113 nonparticipating HCA sites (results not shown). The median hospital size was 177 beds and the median ICU size was 16 beds. The hospitals did not differ between study groups (Table 1). At baseline, 45% of the facilities reported having a CLABSI program and 62% a VAP program.

Baseline Characteristics of the Virtual Collaborative and Toolkit Groups
Hospital Factors at BaselineVirtual CollaborativeToolkitP Value
  • Abbreviations: IQR, interquartile range; SD, standard deviation.

  • One of the 30 hospitals randomized to the Toolkit group was subsequently sold, resulting in 29 hospitals for this study condition.

Number of hospitals3129* 
ICU annual patient volume, median (IQR)568 (294, 904)578 (244, 1077)0.93
ICU patient length of stay days, median (IQR)3882 (1758, 5718)4228 (1645, 6725)0.95
ICU mortality rate, percent (SD)5.7% (3.1%)7.1% (3.6%)0.13
Medicare/Medicaid, percent (SD)68.6% (9.5%)68.5% (10.1%)0.95
Percent admitted to ICU from the Emergency Department (SD)71% (15%)67% (20%)0.27
Percent female (SD)49.7% (5.7%)50.3% (7.7%)0.79
Medicare case‐mix weight, mean (SD)1221 (1007)1295 (1110)0.82
Percent hospitalist ICU management47%40%0.61

The baseline and quarterly median and pooled infection rates for the Toolkit and Collaboration groups are shown in Table 2 for CLABSI and in Table 3 for VAP. There were no significant differences in the baseline rates for either CLABSI (P = 0.24) or VAP (P = 0.72) between the Collaborative and Toolkit groups. There was no significant change for either CLABSI or VAP outcomes at either 12 or 18 months of follow‐up. The median bloodstream infection rate for all participating hospitals was 2.27 at baseline, 1.18 at 12 months (P = 0.13), and 2.23 per 1000 catheter days 18 months later (P = 0.95). The median VAP rate for participating hospitals was 2.90 at baseline, 2.67 at 12 months (P = 0.44), and 2.52 per 1000 ventilator days 18 months later (P = 0.84). The hierarchical regression analysis found that neither the Collaborative nor Toolkit groups improved CLABSI (P = 0.75 and P = 0.83, respectively) or VAP (P = 0.61 and P = 0.37, respectively) rates over time, and there was no differential performance between the 2 groups for either outcome (bloodstream infection, P = 0.71; VAP, P = 0.80).

CLABSI Rates, per 1000 Catheter Days, Overall and by Study Group
 OverallVirtual CollaborativeToolkit
 N = 59 HospitalsN = 30 HospitalsN = 29 Hospitals
Study PeriodHospital Median (IQR)Rate Pooled Across HospitalsHospital Median (IQR)Rate Pooled Across HospitalsHospital Median (IQR)Rate Pooled Across Hospitals
  • Abbreviation: IQR, interquartile range.

Baseline2.27 (0.00‐3.98)2.421.84 (0.00‐3.83)1.672.42 (0.65‐6.80)3.05
3 Month2.27 (1.30‐4.69)2.612.24 (0.54‐4.69)2.342.47 (1.48‐5.35)2.85
6 Month2.37 (0.00‐4.29)2.732.28 (0.00‐3.73)2.352.54 (0.00‐4.98)3.09
9 Month1.66 (0.00‐3.84)2.451.76 (0.00‐3.74)2.281.23 (0.00‐3.93)2.59
12 Month1.18 (0.00‐3.10)2.171.18 (0.00‐2.71)1.721.17 (0.00‐3.61)2.58
15 Month1.93 (0.00‐4.25)2.292.04 (0.00‐4.91)2.531.77 (0.00‐3.30)2.08
18 Month2.23 (0.00‐4.97)2.732.76 (0.00‐4.67)2.751.16 (0.00‐5.46)2.72
VAP Rates per 1000 Ventilator Days, Overall and by Study Group
Study PeriodOverallVirtual CollaborativeToolkit
N = 59 HospitalsN = 30 HospitalsN = 29 Hospitals
Hospital Median (IQR)Rate Pooled Across HospitalsHospital Median (IQR)Rate Pooled Across HospitalsHospital Median (IQR)Rate Pooled Across Hospitals
  • Abbreviation: IQR, interquartile range.

Baseline2.90 (0.00‐6.14)3.972.14 (0.00‐6.09)3.433.49 (0.00‐7.04)4.36
3 Month3.12 (0.00‐8.40)4.463.01 (0.00‐9.11)4.223.32 (0.00‐8.25)4.62
6 Month3.40 (0.00‐7.53)4.972.72 (0.00‐7.09)4.814.61 (0.00‐9.37)5.10
9 Month1.49 (0.00‐4.87)2.990 (0.00‐3.94)2.512.27 (0.00‐6.27)3.36
12 Month2.67 (0.00‐4.60)4.392.67 (0.00‐4.47)3.822.66 (0.00‐4.82)4.95
15 Month3.06 (0.00‐5.10)4.032.40 (0.00‐3.94)3.573.65 (1.15‐6.57)4.45
18 Month2.52 (0.00‐7.45)4.612.93 (0.00‐7.63)5.022.06 (0.00‐6.59)4.31

The poststudy survey was completed by 27 of 31 (87%) of Collaborative group hospitals and 19 of the 29 (66%) Toolkit hospitals. Both groups reported QI improvement efforts to prevent CLABSI (Collaborative 97% vs. Toolkit 88%, P = 0.29) and VAP (Collaborative 97% vs. Toolkit 96%, P = 0.99). Eighty‐three percent of the Collaborative group implemented all components of the bloodstream infection prevention interventions compared with 64% for the Toolkit group (P = 0.13; Figure 1). The Collaborative group implemented daily catheter review more often than the Toolkit group (P = 0.04) and began the process implementation sooner following study implementation (P = 0.006 vs. Toolkit; see Supporting Information Figure). Eighty‐six percent of the Collaborative group implemented the complete VAP prevention interventions vs. 64% of the Toolkit group (P = 0.06; Figure 1) and the Collaborative group conducted the sedation vacation intervention more often (P = 0.03).

Figure 1
(A) Follow‐up survey of self‐reported implementation of key CLABSI prevention interventions by study group. (B) Follow‐up survey of self‐reported implementation of key VAP prevention interventions by study group.

The Collaborative group participated in 57% of the seminars, whereas the Toolkit group participated in 39% (P = 0.014). Members of both groups attended more than half the clinical topics (Collaborative 64% vs. Toolkit 56%, P = 0.37). The Collaborative group had greater participation in the data and method topics (Collaborative 50% vs. Toolkit 22%, P < 0.001). The proportion of hospitals finding the seminars useful to their QI efforts was 49% for the Collaborative and 30% for the Toolkit group (P = 0.017). When restricted to hospitals that participated in the seminars, the usefulness rating was higher for both clinical (91% for the Collaborative and 86% for Toolkit) and Data/Methods (79% for Collaborative and 55% for Toolkit) topics.

A set of 14 tools were produced during the study period (Table 4); 9 clinically related tools (eg, checklists, algorithms, protocols, and flowsheets) and 5 data monitoring and quality improvement tools (eg, easy‐to‐use statistical process control spreadsheet templates, quality improvement tools, and computer tools). The Collaborative group downloaded a median of 10 tools and the Toolkit group a median of 7 (P = 0.051). The groups did not differ in their access to the clinical tools (P = 0.23) but the Collaborative group accessed a greater proportion of the data/methods tools (P = 0.004).

Follow‐up Survey on Study Groups' Tool Use and Strategies for Improvement
Tool Access and StrategiesCollaborative HospitalsaTool Kit HospitalsaP‐value
N = 36 ICUsN = 25 ICUs
  • Post‐survey respondents included 36 ICUs in 26 of the 30 Collaborative Group hospitals and 25 ICUs in 19 of the 29 Tool Kit Group hospitals.

Clinical Tool Use61%49%0.23
BSI Surveillance Guide22/36 (61%)13/25 (52%)0.60
BSI Checklist31/36 (86%)16/25 (64%)0.06
VAP Diagnosis Algorithm24/36 (67%)15/25 (60%)0.60
Ventilator Weaning Protocol23/36 (64%)11/25 (44%)0.18
VAP Surveillance Guide21/36 (58%)12/25 (48%)0.44
VAP Daily Assessment17/36 (47%)6/25 (24%)0.10
Ventilator Weaning Protocol (Flowsheet)15/36 (42%)11/25 (44%)0.99
Data Tools56%30%0.004
QI Implementation Tools19/36 (53%)6/25 (24%)0.03
BSI Statistical Process Control23/36 (64%)5/25 (20%)0.001
VAP Bundle23/36 (64%)11/25 (44%)0.18
VAP Statistical Process Control21/36 (58%)3/25 (12%)0.001
Strategies69%54%0.017
Protocols for BSI24/36 (67%)19/25 (76%)0.57
Protocols for VAP22/36 (61%)9/25 (36%)0.07
Computer Documentation for BSI24/36 (67%)13/25 (52%)0.29
Computer Documentation for VAP25/36 (69%)15/25 (60%)0.58
Increased Staffing3/36 (8%)0/25 (0%)0.26
Written Education for BSI31/36 (86%)19/25 (76%)0.33
Written Education for VAP30/36 (83%)19/25 (76%)0.52
Continuing Education Classes for BSI28/36 (78%)16/25 (64%)0.26
Continuing Education Classes for VAP30/36 (83%)17/25 (68%)0.21
QI teams27/36 (75%)14/25 (56%)0.16
Provider Performance Feedback for BSI23/36 (64%)11/25 (44%)0.18
Provider Performance Feedback for VAP24/36 (67%)11/25 (44%)0.11
Implementation of BSI Checklist28/36 (78%)15/25 (60%)0.16
Implementation of VAP Checklist31/36 (86%)13/25 (52%)0.007

Both groups relied primarily on implementation of protocols and informatics approaches (Table 4) without increasing staff levels. The predominant strategy was education; both groups provided written educational materials and classes to their providers. There was a trend for more Collaborative group members to implement QI teams (Table 4, P = 0.16 compared with the Toolkit group). Although the preponderance of both groups provided feedback reports to their hospital leaders and unit managers, Collaborative group hospitals showed a trend for providing feedback to front‐line providers (P = 0.11). With respect to self‐reported interventions, 78% of the Collaborative ICUs reported implementing a CLABSI checklist and 86% a VAP checklist, whereas only 60% of the Toolkit group reported implementation of a CLABSI checklist (P = 0.16) and 52% a VAP checklist (P = 0.007). Once a tool was implemented, both groups reported a high rate of sustaining the implementation (ranging from 86% to 100%). There also seemed to be a pattern of sequencing the interventions. Initial efforts tend to focus on provider education and evidence‐based protocols. Later efforts include more formal formation of QI teams followed by implementation of checklists. The evidence for sequencing of interventions is qualitative; we lacked subgroup sample size to substantiate these results with statistical analysis.

Discussion

In our investigation of Virtual Collaborative and Toolkit strategies for spreading the implementation of safe practices for CLABSI and VAP, ICUs in the Collaborative group had more complete implementation of the processes for prevention of hospital‐associated infections. Although both groups accessed clinical resources consistent with surveillance and clinical education, the Virtual Collaborative group attended to data and implementation methods more likely to lead to systemic CQI and organizational changes. ICUs that engaged these resources believed them useful in implementing QI, and more than 85% of the practices were sustained once integrated into routine care. Although the Collaborative ICUs were about 50% more likely to implement improvement strategies, these differences in implementation and process of care did not translate into group differences or longitudinal changes in infection rates.

In contrast to the context of our investigation, most published QI studies on health careassociated infection prevention report high baseline rates followed by a significant decline in infection rates.1419 The baseline infection rates in our study hospitals were actually below the endpoint found in many prior studies, suggesting that any marginal effects from our intervention would be more difficult to detect. Our study was implemented during the IHI's 100,000 Lives Campaign,20 a trend that may have brought about these lower baseline rates and thus a tighter margin for improvement.

The median CLABSI baseline rate in the well‐publicized Michigan hospital study was 2.7 per 1000 catheter days.21, 22 Although our baseline rate was similar (2.27 per 1000 catheter days), their reported postintervention rate was near zero, inferring nearly total elimination of the risk for CLABSI within 3‐18 months of study implementation. Several other studies using a collaborative approach have similarly reported high‐performance near‐zero results in reducing VAP23, 24 and CLABSI2528 rates. The difference between the present and previously published near‐zero result outcomes raises questions about collaboration‐based studies. We noticed 2 phenomena. First, there was slow uptake of data‐driven QI, and second, there was a differential uptake between general knowledge (clinical evidence and education) and QI implementation knowledge.29, 30

Lack of infrastructure to support data‐driven QI remains a significant barrier throughout the health care system, and teams in collaboratives often must work intensively toward improving their information systems' capability for the purpose of data‐driven decision support.1, 15, 31, 32 Systematic, standardized collection of CLABSI and VAP outcomes was initially lacking in many of our study hospitals,10 and our project expended early effort to deploy a system‐wide standardized infection control database registry.

Both of our study groups gravitated toward educational training and evidence‐based protocol decision‐support strategies. A focus only on established surveillance and education‐based fixes (eg, asking clinicians to follow a protocol within their existing care processes) have produced 32% to 57% reductions in health careacquired infections.3335 These early gains, however, are unlikely to produce the sustained near‐zero results that some collaborative teams have reported.22, 25

The ability to achieve sustained high‐performance results depends on organizational context and requires time.31 A potential benefit of collaboratives might be the return on investment attained by organizational change in quality and safety climate and its influence across the whole organization.19, 31, 36 Participants requiring systems training in the CQI process may not gain these benefits until well into their collaborative.31 For example, accumulating evidence demonstrates that the use of checklists can reduce errors of omission. Although a checklist seems a simple intervention, its effective implementation into routine care processes actually requires time for system redesign that addresses changes in multidisciplinary roles and responsibilities, frontline clinician and mid‐level management buy‐in, new methods of data collection and feedback, unanticipated involvement of ancillary services (eg, medical records, housekeeping), as well as changes to organizational policies, expectations, and priorities that connect silos of care and integrate hierarchical operations. Wall et al.37 and Pronovost and colleagues19, 21, 22, 25 highlighted the strategic effectiveness of embedding a checklist as a behavioral and data collection tool into frontline care process, leading to a redefined role of nursing, as well as new data for further cycles of improvement that collectively reduced infection rates. In our study, the Virtual Collaborative group did not have greater use of CLABSI and VAP checklists until the QI teams had been formed months into the project, consistent with the hypothesis that beneficial translation of desired changes in process of care to observed improvements in patient outcomes may take longer than 18 months to achieve19, 25, 27, 38 as opposed to the remarkable 3 months reported in the Keystone ICU project.21

Our study has several limitations. Our intervention did not mandate fixed specific components of intervention or QI methods. Each medical center was free to tailor its use of tools and change ideas, producing site variation in implementation methods and investment in support of QI. Like other multicomponent, multidimensional intervention studies, we were not able to test the effectiveness of particular QI components or the thoroughness of surveillance for CLABSI and VAP related to efforts to standardize the approach, and we did not have the resources to monitor the intensity with which participants approached QI. Furthermore, our data were dependent on self‐reports and were not verified by independent assessment of the fidelity with which the interventions were implemented, a checklist was embedded into usual care, or practices were enforced by nurses. In addition, the virtual collaborative circumvents the face‐to‐face learning sessions that might play a role in collaborative social networking, peer pressure, and acculturation.31, 36

Despite these limitations, we found that the Virtual Collaborative performed just like a Breakthrough Collaborative with a gradual uptake of implementation science using QI methods, team management, and statistical process control tools. The Toolkit condition had an even slower uptake. From an organization's perspective, the bottom‐line decision is whether a greater and meaningful proportion of collaborative participants will be successful to justify the investment of effort compared to a toolkit‐only approach. Our findings suggest that organizations engaged in change but lacking expertise in implementation science can potentially benefit from the acculturation, experiential learning, and uptake of QI provided by a collaborative.

In summary, although our Virtual Collaborative intervention was more likely to produce changes in ICU processes of care, there were no improvements in patient outcomes over this 18‐month study. The current popularity of evidence‐based guidelines, care protocols, prevention awareness, and surveillance may have produced a background of secular trend, making it difficult to ascertain effects of our QI intervention. Nonetheless, important lessons can be gleaned from this randomized controlled trial. Our study supports the proposition that as long as organizations vary in their capacity for and commitment to the science of QI and systems engineering, we should anticipate variation, uncertainty, and mixed results from short‐term, rapid cycle initiatives.27, 28, 31, 32, 39, 40 The untested, longer‐term benefit produced by a collaborative may be its stimulation of enduring systems engineering that optimizes an environment for QI of health care processes focused on desired outcomes.

Acknowledgements

The authors thank the Agency for Healthcare Research and Quality collaborative investigators for their work in this study: Xu Lei Liu, MS, at Vanderbilt; Laurie Brewer, RN MBA, Jason Hickok, Steve Horner, Susan Littleton, Patsy McFadden, RN BSN MPA CIC, Steve Mok, PharmD, Jonathan Perlin, MD PhD, Joan Reischel, RN BSN CCRN, and Sheri G. Chernestky Tejedor, MD, and all the HCA medical centers that participated in this project.

References
  1. Shortell SM,Bennett CL,Byck GR.Assessing the impact of continuous quality improvement on clinical practice: What it will take to accelerate progress.Milbank Q.1998;76:593624.
  2. Berwick DM.Continuous improvement as an ideal in health care.N Engl J Med.1989;320:5356.
  3. Kilo CM.A framework for collaborative improvement: Lessons from the Institute for Healthcare Improvement's Breakthrough Series.Qual Manag Health Care.1998;6(4):113.
  4. Ayers LR,Beyea SC,Godfrey MM,Harper DC,Nelson EC,Batalden PB.Quality improvement learning collaboratives.Qual Manag Health Care.2005;14:234237.
  5. Boushon B,Provost L,Gagnon J,Carver P.Using a virtual breakthrough series collaborative to improve access in primary care.Jt Comm J Qual Patient Saf.2006;32:573584.
  6. Eagle KA,Gallogly M,Mehta RH, et al.Taking the national guideline for care of acute myocardial infarction to the bedside: Developing the guideline applied in practice (GAP) initiative in Southeast Michigan.Jt Comm J Qual Improv.2002;28:519.
  7. Adams K,Corrigan JM.Priority Areas for National Action: Transforming Health Care Quality.Washington, DC:The National Academies Press;2003.
  8. Greevy RA,Lu B,Silber SH,Rosenbaum P.Optimal multivariate matching before randomization.Biostatistics.2004;5:263275.
  9. Institute for Healthcare Improvement. The 100,000 lives campaign. http://www.ihi.org/IHI/Programs/Campaign.htm;2005.
  10. Talbot T,Tejedor SC,Greevy RA, et al.Survey of infection control programs in a large, national healthcare system.Infect Control Hosp Epidemiol.2007;28:14011403.
  11. Shojania KG,McDonald KM,Wachter RM,Owens DK. Closing the quality gap: A critical analysis of quality improvement strategies, Volume 1‐Series overview and methodology. Technical Review 9 (Contract No 290–02‐0017 to the Stanford University‐UCSF Evidence‐based Practices Center), 2004. www.ahrq.gov/clinic/tp/qgap1tp.htm. Accessed November 11,2010.
  12. Mohr JJ,Batalden PB.Improving safety on the front lines: the role of clinical microsystems.Qual Saf Health Care.2002;11:4550.
  13. Borenstein M,Rothstein H,Cohen J.SamplePower 2.0.Chicago, IL:SPSS Inc.;2001.
  14. Berriel‐Cass D,Adkins FW,Jones P,Fakih MG.Eliminating nosocomial infections at Ascension Health.Jt Comm J Qual Patient Saf.2006;32:612620.
  15. Bonello RS,Fletcher CE,Becker WK, et al.An intensive care unit quality improvement collaborative in nine department of Veterans Affairs hospitals: reducing ventilator‐associated pneumonia and catheter‐related bloodstream infection rates.Jt Comm J Qual Patient Saf.2008;34:639645.
  16. Cocanour CS,Peninger M,Domonoske BD, et al.Decreasing ventilator‐associated pneumonia in a trauma ICU.J Trauma.2006;61:122130.
  17. Frankel HL,Crede WB,Topal JE,Roumanis SA,Devlin MW,Foley AB.Use of corporate six sigma performance‐improvement strategies to reduce incidence of catheter‐related bloodstream infections in a surgical ICU.J Am Coll Surg.2005;201:349358.
  18. Jain M,Miller L,Belt D,King D,Berwick DM.Decline in ICU adverse events, nosocomial infections and cost through a quality improvement initiative focusing on teamwork and culture change.Qual Saf Health Care.2006;15:235239.
  19. Resar R,Pronovost PJ,Harden C,Simmonds R,Rainey T,Nolan TW.Using a bundle approach to improve ventilator care processes and reduce ventilator‐associated pneumonia.Jt Comm J Qual Patient Saf.2005;31:243248.
  20. Berwick DM,Calkins DR,McCannon CJ,Hackbarth AD.The 100000 lives campaign: Setting a goal and a deadline for improving health care quality.JAMA.2006;295:324327.
  21. Pronovost PJ,Needham D,Berenholtz SM.An intervention to decrease catheter‐related bloodstream infections in the ICU.N Engl J Med.2006;355:27252732.
  22. Pronovost PJ,Berenholtz SM,Goeschel C, et al.Improving patient safety units in Michigan.J Crit Care.2008;23:207221.
  23. Fox MY.Toward a zero VAP rate: Personal and team approaches in the ICU.Crit Care Nurs Q.2006;29:108114.
  24. Youngquist P,Carroll M,Farber M, et al.Implementing a ventilator bundle in a community hospital.Jt Comm J Qual Patient Saf.2007;33:219225.
  25. Berenholtz SM,Pronovost PJ,Lipsett PA, et al.Eliminating catheter‐realted bloodstream infections in the intensive care unit.Crit Care Med.2004;32:20142020.
  26. Harnage S.Innovative bundle wipes out catheter‐related bloodstream infections.Nursing.2008;38:1718.
  27. Koll BS,Straub TA,Jalon HS,Block R,Heller KS,Ruiz RE.The CLABs collaborative: a regionwide effort to improve the quality of care in hospitals.Jt Comm J Qual Patient Saf.2008;34:713723.
  28. Render ML,Brungs S,Kotagal U, et al.Evidence‐based practice to reduce central line infections.Jt Comm J Qual Patient Saf.2006;32:253260.
  29. Nembhard IM.Learning and improving in quality improvement collaboratives: which collaborative features do participants value most?Health Serv Res.2009;44(2 Pt 1):359378.
  30. Grossman E,Keegan T,Lesser AL, et al.Inside the health disparities collaboratives: a detailed exploration of quality improvement at community health centers.Med Care.2008;46:489496.
  31. Ovretveit J,Bate P,Cleary P, et al.Quality collaboratives: lessons from research.Qual Saf Health Care.2002;11:345351.
  32. Pearson ML,Wu S,Schaefer CT, et al.Assessing the implementation of the chronic care model in quality improvement collaboratives.Health Serv Res.2005;40:978996.
  33. Gastmeier P,Geffers C.Prevention of ventilator‐associated pneumonia: Analysis of studies published since 2004.J Hosp Infect.2007;67:18.
  34. McKinley LL,Moriarty HJ,Short TH,Johnson CC.Effect of comparative data feedback on intensive care unit infection rates in a Veterans Administration Hospital network system.Am J Infect Control.2003;31:397404.
  35. Salahuddin N,Zafar A,Sukhyani L, et al.Reducing ventilator‐associated pneumonia rates through a staff education programme.J Hosp Infect.2004;57:223227.
  36. Alexander JA,Weiner BJ,Shortell SM,Baker LC.Does quality improvement implementation affect hospital quality of care?Hosp Top.2007;85:312.
  37. Wall RJ,Ely EW,Ellis D,Dittus RS,Foss J,Speroff T.Using real‐time process measurements to reduce catheter‐related bloodstream infections in the intensive care unit.Qual Saf Health Care.2005;14:295302.
  38. Esmail R,Duchscherer G,Giesbrecht J,King J,Ritchie P,Zuege D.Prevention of ventilator‐associated pneumonia in the Calgary health region: a Canadian success story!Healthcare Qual.2008;11(3 Spec No):129136.
  39. Mittman BS.Creating the evidence base for quality improvement collaboratives.Ann Intern Med.2004;140:897901.
  40. Schouten LMT,Hulscher MEJL,Everdigen JJE,Huijsman R,Grol RPTM.Evidence for the impact of quality improvement collaboratives: systematic review.Br Med J.2008;336:14911494.
References
  1. Shortell SM,Bennett CL,Byck GR.Assessing the impact of continuous quality improvement on clinical practice: What it will take to accelerate progress.Milbank Q.1998;76:593624.
  2. Berwick DM.Continuous improvement as an ideal in health care.N Engl J Med.1989;320:5356.
  3. Kilo CM.A framework for collaborative improvement: Lessons from the Institute for Healthcare Improvement's Breakthrough Series.Qual Manag Health Care.1998;6(4):113.
  4. Ayers LR,Beyea SC,Godfrey MM,Harper DC,Nelson EC,Batalden PB.Quality improvement learning collaboratives.Qual Manag Health Care.2005;14:234237.
  5. Boushon B,Provost L,Gagnon J,Carver P.Using a virtual breakthrough series collaborative to improve access in primary care.Jt Comm J Qual Patient Saf.2006;32:573584.
  6. Eagle KA,Gallogly M,Mehta RH, et al.Taking the national guideline for care of acute myocardial infarction to the bedside: Developing the guideline applied in practice (GAP) initiative in Southeast Michigan.Jt Comm J Qual Improv.2002;28:519.
  7. Adams K,Corrigan JM.Priority Areas for National Action: Transforming Health Care Quality.Washington, DC:The National Academies Press;2003.
  8. Greevy RA,Lu B,Silber SH,Rosenbaum P.Optimal multivariate matching before randomization.Biostatistics.2004;5:263275.
  9. Institute for Healthcare Improvement. The 100,000 lives campaign. http://www.ihi.org/IHI/Programs/Campaign.htm;2005.
  10. Talbot T,Tejedor SC,Greevy RA, et al.Survey of infection control programs in a large, national healthcare system.Infect Control Hosp Epidemiol.2007;28:14011403.
  11. Shojania KG,McDonald KM,Wachter RM,Owens DK. Closing the quality gap: A critical analysis of quality improvement strategies, Volume 1‐Series overview and methodology. Technical Review 9 (Contract No 290–02‐0017 to the Stanford University‐UCSF Evidence‐based Practices Center), 2004. www.ahrq.gov/clinic/tp/qgap1tp.htm. Accessed November 11,2010.
  12. Mohr JJ,Batalden PB.Improving safety on the front lines: the role of clinical microsystems.Qual Saf Health Care.2002;11:4550.
  13. Borenstein M,Rothstein H,Cohen J.SamplePower 2.0.Chicago, IL:SPSS Inc.;2001.
  14. Berriel‐Cass D,Adkins FW,Jones P,Fakih MG.Eliminating nosocomial infections at Ascension Health.Jt Comm J Qual Patient Saf.2006;32:612620.
  15. Bonello RS,Fletcher CE,Becker WK, et al.An intensive care unit quality improvement collaborative in nine department of Veterans Affairs hospitals: reducing ventilator‐associated pneumonia and catheter‐related bloodstream infection rates.Jt Comm J Qual Patient Saf.2008;34:639645.
  16. Cocanour CS,Peninger M,Domonoske BD, et al.Decreasing ventilator‐associated pneumonia in a trauma ICU.J Trauma.2006;61:122130.
  17. Frankel HL,Crede WB,Topal JE,Roumanis SA,Devlin MW,Foley AB.Use of corporate six sigma performance‐improvement strategies to reduce incidence of catheter‐related bloodstream infections in a surgical ICU.J Am Coll Surg.2005;201:349358.
  18. Jain M,Miller L,Belt D,King D,Berwick DM.Decline in ICU adverse events, nosocomial infections and cost through a quality improvement initiative focusing on teamwork and culture change.Qual Saf Health Care.2006;15:235239.
  19. Resar R,Pronovost PJ,Harden C,Simmonds R,Rainey T,Nolan TW.Using a bundle approach to improve ventilator care processes and reduce ventilator‐associated pneumonia.Jt Comm J Qual Patient Saf.2005;31:243248.
  20. Berwick DM,Calkins DR,McCannon CJ,Hackbarth AD.The 100000 lives campaign: Setting a goal and a deadline for improving health care quality.JAMA.2006;295:324327.
  21. Pronovost PJ,Needham D,Berenholtz SM.An intervention to decrease catheter‐related bloodstream infections in the ICU.N Engl J Med.2006;355:27252732.
  22. Pronovost PJ,Berenholtz SM,Goeschel C, et al.Improving patient safety units in Michigan.J Crit Care.2008;23:207221.
  23. Fox MY.Toward a zero VAP rate: Personal and team approaches in the ICU.Crit Care Nurs Q.2006;29:108114.
  24. Youngquist P,Carroll M,Farber M, et al.Implementing a ventilator bundle in a community hospital.Jt Comm J Qual Patient Saf.2007;33:219225.
  25. Berenholtz SM,Pronovost PJ,Lipsett PA, et al.Eliminating catheter‐realted bloodstream infections in the intensive care unit.Crit Care Med.2004;32:20142020.
  26. Harnage S.Innovative bundle wipes out catheter‐related bloodstream infections.Nursing.2008;38:1718.
  27. Koll BS,Straub TA,Jalon HS,Block R,Heller KS,Ruiz RE.The CLABs collaborative: a regionwide effort to improve the quality of care in hospitals.Jt Comm J Qual Patient Saf.2008;34:713723.
  28. Render ML,Brungs S,Kotagal U, et al.Evidence‐based practice to reduce central line infections.Jt Comm J Qual Patient Saf.2006;32:253260.
  29. Nembhard IM.Learning and improving in quality improvement collaboratives: which collaborative features do participants value most?Health Serv Res.2009;44(2 Pt 1):359378.
  30. Grossman E,Keegan T,Lesser AL, et al.Inside the health disparities collaboratives: a detailed exploration of quality improvement at community health centers.Med Care.2008;46:489496.
  31. Ovretveit J,Bate P,Cleary P, et al.Quality collaboratives: lessons from research.Qual Saf Health Care.2002;11:345351.
  32. Pearson ML,Wu S,Schaefer CT, et al.Assessing the implementation of the chronic care model in quality improvement collaboratives.Health Serv Res.2005;40:978996.
  33. Gastmeier P,Geffers C.Prevention of ventilator‐associated pneumonia: Analysis of studies published since 2004.J Hosp Infect.2007;67:18.
  34. McKinley LL,Moriarty HJ,Short TH,Johnson CC.Effect of comparative data feedback on intensive care unit infection rates in a Veterans Administration Hospital network system.Am J Infect Control.2003;31:397404.
  35. Salahuddin N,Zafar A,Sukhyani L, et al.Reducing ventilator‐associated pneumonia rates through a staff education programme.J Hosp Infect.2004;57:223227.
  36. Alexander JA,Weiner BJ,Shortell SM,Baker LC.Does quality improvement implementation affect hospital quality of care?Hosp Top.2007;85:312.
  37. Wall RJ,Ely EW,Ellis D,Dittus RS,Foss J,Speroff T.Using real‐time process measurements to reduce catheter‐related bloodstream infections in the intensive care unit.Qual Saf Health Care.2005;14:295302.
  38. Esmail R,Duchscherer G,Giesbrecht J,King J,Ritchie P,Zuege D.Prevention of ventilator‐associated pneumonia in the Calgary health region: a Canadian success story!Healthcare Qual.2008;11(3 Spec No):129136.
  39. Mittman BS.Creating the evidence base for quality improvement collaboratives.Ann Intern Med.2004;140:897901.
  40. Schouten LMT,Hulscher MEJL,Everdigen JJE,Huijsman R,Grol RPTM.Evidence for the impact of quality improvement collaboratives: systematic review.Br Med J.2008;336:14911494.
Issue
Journal of Hospital Medicine - 6(5)
Issue
Journal of Hospital Medicine - 6(5)
Page Number
271-278
Page Number
271-278
Publications
Publications
Article Type
Display Headline
Quality improvement projects targeting health care–associated infections: Comparing virtual collaborative and toolkit approaches
Display Headline
Quality improvement projects targeting health care–associated infections: Comparing virtual collaborative and toolkit approaches
Legacy Keywords
patient safety, quality improvement, central line–associated bloodstream infection, ventilator‐associated pneumonia
Legacy Keywords
patient safety, quality improvement, central line–associated bloodstream infection, ventilator‐associated pneumonia
Sections
Article Source

Copyright © 2011 Society of Hospital Medicine

Disallow All Ads
Correspondence Location
Department of Medicine, Center for Health Services Research, 6000 Medical Center East, Vanderbilt University School of Medicine, Nashville, TN 37232
Content Gating
No Gating (article Unlocked/Free)
Alternative CME
Article PDF Media