This article provides a reply to a Wall Street Journal article entitled: “Medicare Shift Fails to Cut Hospital Infections” and posted October 10, 2012. It provides some tools to use when reading and evaluating the scientific merit and implications of published studies claiming significant reductions in infection rates and/or the significant impact of an evaluated intervention intended to prevent infections in ICUs (intensive care units) and other healthcare settings. These studies are routinely used to evaluate the quality of health care and to allocate limited resources to specific healthcare settings.
Background
A number of news sources report today on the results of a study by Lee et al (2012) that was just published in “The New England Journal of Medicine.”
Lee et al.’s article – which can be read here evaluates the impact on the infection rate of the Centers for Medicare and Medicaid Services’ (CMS) policy to reduce payments for certain types of healthcare-associated infections (e.g., central catheter–associated bloodstream infections).
For example, Lee et al. study was reported by the Wall Street Journal (“Medicare Shift Fails to Cut Hospital Infections”) or in a Boston newspaper article (“Penalty program had little effect on hospital-acquired infections”) and, too, in a Kaiser Heath News report (“CMS Penalties Don’t Change Hospital-Acquired Infection Rates”).
Findings
Lee et al.’s (2012) study concludes that a high-profile Medicare policy intended to reduce certain hospitals-associated infections had no impact on the rates of, for example, central line associated bloodstream infections (CLABSIs).
Namely, Lee et al.’s abstract states that: “We found no evidence that the 2008 CMS policy to reduce payments for central catheter–associated bloodstream infections and catheter-associated urinary tract infections had any measurable effect on infection rates in U.S. hospitals.”
Centers for Medicare and Medicaid Services (CMS)
Lee et al.’s study uses data from approximately 400 hospitals that were reported between January 2006 to Match 2011, and compares the infection rates before and after implementation of CMS’s policy of financial penalties.
According to CMS (as reported by the Wall Street Journal), “’Taken all together, our policies (CMS’s) are working’ to reduce infections patients get while in the hospital. The agency pointed to other efforts, including one program funded by the federal Agency for Healthcare Research and Quality that was tied to a 40% decrease in the rate of the catheter-linked bloodstream infections in hospital intensive-care units.”
The study CMS is referring to is discussed in a press release, which states that:
“The near elimination of central line–associated bloodstream infections (CLABSIs) in intensive care units (ICUs) in Michigan stands as one of the landmark accomplishments of the patient safety field. Although the checklist for CLABSI prevention has been widely publicized, equally important components of the intervention included the comprehensive unit-based safety program (CUSP) and interventions to improve safety culture in participating ICUs. The Agency for Healthcare Research and Quality subsequently sponsored an effort to extend the success of the Michigan initiative nationwide, centered around implementation of the CUSP.”
Not So Fast
This press release adds that: “The initial results (of this effort) indicate another remarkable success, with CLABSI rates being reduced by 40% across 1100 participating ICUs. It is notable that these reductions were accomplished even though the baseline rate of CLABSI was already lower than in prior studies.”
The study to which this press release refers, however, is not of a design that permits the cause-and-effect conclusion that this effort, or any effort, reduced the infection rate by a quantitative amount, namely, by 40% – this press release having drawn such a conclusion notwithstanding.
Rather, the press release and original study are limited to concluding only an “association” between the federally funded initiative (that uses an evidence-based tool kit, including checklists) and to measuring a qualitative reduction in the infection rate. For more on this topic, read by peer-reviewed article and review of the literature here.
Medical journals
The CDC is not exempt from making these same types of missteps and from drawing potentially unsound conclusions about trends in infection rates, based on infection data that have not been validated and may be inaccurate, unintentionally biases, incomplete, and unreliable.
In addition to The New England Journal of Medicine, other medical journals, too, including the CDC’s MMWR, have routinely publish studies that advance conclusions that the designs of the studies do not permit. Such an error is significant and may be called a “post hoc fallacy,” which derives its name from the Latin phrase: “Post hoc, ergo propter hoc.”
Indeed, it can be reasonably argued that failure to validate the infection data used by a study to measure the effectiveness of an intervention on the infection rate significantly limits the study’s scope, breadth, and contribution. (Note that Lee et al.’s study does not discuss the importance of data validation.)
Dr. Muscarella has written a number of articles that discuss published infection rates vis-a-vis the “post hoc fallacy” and the common practice of renown medical journals publishing articles that advance conclusions based on infection data that: (i) have not been validated and, therefore, are subject to doubt; and (ii) are collected using non-randomized, prospective cohort studies that, limited by their design to associations between two variables (i.e., an intervention and a measured reduction in the infection rate) frequently advance cause-and-effect relationships:
Read Dr. Muscarella’s two related articles “Dear Pediatrics” and “Assessment of the Reported Effectiveness of Five Different Quality- Improvement Initiatives for the Prevention of Central Line-Associated Bloodstream Infections in Intensive Care Units.”
Entitled “Review of a CDC Report about Healthcare-Associated Infections,” also read Dr. Muscarella’s review, if critique, of a Vital Signs study that the Centers for Disease Control and Prevention published in Morbidity and Mortality Weekly Report about central line-associated bloodstream infections.
That studies eliciting potentially high-profile news headlines about infection rates and the quality of health care use infection data have been validated for accuracy and completeness (or, otherwise, disclosure in their findings that their infection data have not been validated, to ensure no misunderstandings) cannot be understated.
Article by: Lawrence F Muscarella, PhD. (Disclosure: Dr. Muscarella – who can be followed on Twitter @MuskiePhD – has no financial ties whatsoever, direct or indirect, understood or implied, with the study of, or with any products associated with, central line infections in intensive care units.)