Published Rates of Infections in ICUs: More Conjectural than Scientific?

December 9, 2013 — Every week, a newspaper article, federal report, or scientific study discusses the quality of health care in the U.S.^1-17

Many of these publications focus on efforts to prevent healthcare-associated infections (HAIs), often in intensive care units (ICUs). Recently enacted state laws that mandate the reporting of certain types of HAIs are examples of such efforts.^1,5,6,8,9

The use of patient outcomes to measure the success of many of these efforts is of considerable interest and debate. Although some recent reports suggest that hospitals in the U.S. may be “getting safer,”⁷ others have found the quality of health care to be lacking.^14,15

A federal study by Schaeffer et al. (2010), for example, found infection-control lapses among dozens of inspected ambulatory surgical centers (ASCs) to be common.¹⁴

Read Dr. Muscarella’s accompanying peer-reviewed article “Assessment of the Reported Effectiveness of Five Different Quality-Improvement Initiatives for the Prevention of Central Line-Associated Bloodstream Infections in Intensive Care Units” by clicking here.

Raising additional questions about the quality of health care in the U.S., Landrigan et al. (2010) found “harms” to patients to be similarly common among 10 randomly-selected hospitals in North Carolina, with “little evidence” of improvement (over the 6-year period ending in 2007).¹⁵

Examples of such harming events discussed by Landrigan et al. (2010) are “CLABSIs,” which are a type of HAI and an acronym for bloodstream infections associated with central venous catheters (or “central lines”;  see: Box A).

According to Landrigan et al. (2010), their findings (that harming events were common) “validate” concerns about the quality of health care, at least in hospitals in North Carolina.¹⁵

Their findings also underscore the importance of developing more effective strategies to prevent CLABSIs and other types of HAIs. A growing number of such strategies are based on models of greater accountability and transparency, namely, on rating schemes that grade and rank the quality of hospitals.

Such rating schemes facilitate convenient (although not necessarily valid) comparisons of, in addition to other metrics,¹³ HAI rates among hospitals in the same or different cities, states and countries,  all but compelling hospitals to become more competitive, to improve their care, and to report reduced infection rates.^6,10,12,16   (That such comparisons of  infection data are not limited to hospitals but may also apply to other types of medical facilities, including ASCs, is noted.)

Read Dr. Muscarella’s OP/ED article in The Seattle Times newspaper on this same topic by clicking here.

CLABSIs

Articles, reports and studies frequently laud a hospital’s claimed safety, or publicize how effectively a specific intervention, checklist, or bundle of best practices might have reduced the rates of CLABSIs in adult or pediatric ICUs.^{1-7,10-12,17-24}

CLABSI rates in ICUs have become a popular patient outcome used by consumers, federal and state agencies, and both public and private insurers to assess, rate, and compare the quality of hospitals.^{1,6,8-12,25-32}

According to one report, the public reporting of CLABSI and other HAI rates will help to “save countless lives and dollars.”²⁵

A hospital’s public reporting of a low CLABSI rate might imply to the consumer the safety not only of its ICUs, but also of the hospital’s other departments, including its gastrointestinal (GI) endoscopy and respiratory therapy departments.

A hospital reporting a relatively high, if uncompetitive, CLABSI rate, however, may be viewed as potentially unsafe and be publicly labeled by a consumer magazine as a “poor performer”¹²—a moniker that, whether fair or not, could cause the hospital to experience consumer backlash and  encounter financial, legal, and accreditation hardships.

A first-of-its-kind CDC report

The first of its kind, the Centers for Disease Control and Prevention (CDC) published a report in June (2010) that summarizes CLABSI data reported by more than 1500 (short stay, acute-care) hospitals in 17 (U.S.) states^6,7 whose laws mandate the reporting of CLABSI data to a network within the CDC (see: Box B).

This report found that the number of CLABSIs provided by almost two thirds (n=11) of these states was significantly (18%) fewer than “predicted,”⁶ which the CDC suggests demonstrates that care in hospitals is becoming “safer”⁷ (notwithstanding the concerning findings of Schaeffer et al.’s [2010]¹⁴ study and Landrigan et al.’s [2010] report¹⁵).

Direct hospital comparisons

Although the CDC notes in this state-specific report that its CLABSI data are “not put forth … for direct comparisons between states,”⁶ this report’s formal, impressive, and easy-to-read listing of infection data certainly facilitates, if not ensures, just that:  direct comparisons of CLABSI data (in different states).^{4,5,8-10,12,29,30}

Comparisons of the CLABSI data listed in this state-specific report are similar to those encouraged by Consumers Union, whose March (2010) issue of Consumer Reports lists the CLABSI data reported by 43 hospitals in 10 states.¹²

This magazine urges consumers to “protect” themselves and (when given a choice) select a hospital it labels a “top performer”¹² for having reported a comparatively low CLABSI rate.*

Read Dr. Muscarella’s related article, “Dear CDC,” which reviews a study by the CDC that concludes, based on estimated reductions in the rates of CLABSIs, that the quality of health care in the U.S. is improving.

Data validation

As with any metric or patient outcome used to evaluate a hospital’s quality, the reporting of infection rates can only be as valid as the measured data are accurate.

That these measured CLABSI rates—which may be reported by hospitals to the state’s department of health, a hospital association, web-based software, or the CDC, either voluntarily or as mandated by a state law^6,8,9,30-32—be accurate, therefore, is necessary and underscored, because, in addition to these rates being used to compare hospitals,^6,12,32 both public and private insurers may condition reimbursements on hospitals tracking and reporting CLABSI rates.

Examples of such incentivizing programs include value-based purchasing,^26,33 pay for performance,^10,12,33,34 and the Centers for Medicare and Medicaid Services’ (CMS) pay-for-reporting program.^25,27,28

Otherwise, if reported CLABSI rates have not been validated, then

• the public’s presumption of their accuracy might be incorrect;
• the use of these rates to compare hospitals might not be “credible”;³²
• claims that these rates demonstrate improved health care might be in error;⁷
• instructions urging consumers to use these rates to select a “top” performing hospital might be unsound;¹²
• state laws mandating the reporting of these rates would be arguably  incomplete;^8,9,30-32
• payments or reimbursements to hospitals based on these rates could be problematic;^25-28 and
• conclusions about the effectiveness of an intervention designed to reduce CLABSI rates in ICUs might be flawed.^17-24

Purpose, methodology

This article aims to reconcile reports suggesting health care in U.S. hospitals is becoming safer^6,7 with dissimilar reports that found concerning lapses and harms to be common.^14,15

Specifically, this article’s primary aim is to assess:

• the validity of reported CLABSI rates, which may be used by consumers, governmental agencies, and insurers to evaluate and compare a hospital’s relative safety, or to incentivize improved health care;^26-28,30-32 and
• the completeness of state laws mandating the reporting of these infection rates.^8,9,30-32 As part of this aim, this article seeks to confirm that the CLABSI data listed in  the  CDC’s  aforementioned state-specific report⁶  and in Consumer Reports’ March (2010) issue¹² have been validated.

This article’s secondary aim is to review and assess the soundness of several published prospective cohort studies that conclude that an intervention, collaborative, checklist, or bundle of best-practice strategies reduced the rate of CLABSIs in one or more ICUs by a specific percentage.^12,17-24

These aims were achieved by reviewing several newspaper articles, state laws, federal reports, prospective cohort studies and, among other publications, both the CDC’s state-specific report and Consumer Reports’ article about CLABSIs in its March (2010) issue. (Note: This article’s discussion may also be applied to the prevention of HAIs in other types of health care settings, including GI endoscopy units and ASCs.)

Results

This review questions the validity of the majority of reported rates of CLABSIs. With only a few exceptions, reported CLABSI rates—including those listed in the CDC’s state-specific report⁶ and in Consumer Reports’ article about CLABSIs in its March (2010) issue¹²—have not been validated for accuracy and completeness.^6,8,9,30,32

This finding suggests that the ubiquitous use of reported CLABSI rates to evaluate and compare the relative safety and quality of hospitals by consumers; by the CDC and other federal and governmental agencies; by Consumer Union; and by public and private insurers, among others, may be unsound.^14,15

Read Dr. Muscarella’s “Dear Pediatrics” review of a study in the journal Pediatrics that assesses the quality of care based on estimated reductions in the rates of CLABSIs in neonatal and pediatric intensive care units.

Advancing the use of data that may be in error, the CLABSI data listed in the CDC’s state-specific report were not validated in the majority (n = 12) of the 17 listed states—even though the CDC used these rates to conclude that care in hospitals is getting safer.⁷ (According to this CDC report, the laws in only the remaining 5 states require that the reported CLABSI rates be validated for accuracy and completeness).⁶

In addition, the CLABSI rates reported by more than half (n = 23) of the 43 hospitals listed in the Consumer Reports’ article about CLABSIs similarly were not validated—even though Consumers Union used these rates to provide advice, grade hospitals, and to label some poor or top performers.¹²

Several prospective cohort studies were identified during this review that evaluate how effectively an intervention, checklist, collaborative, or bundle of practices might reduce CLABSI rates in adult or pediatric ICUs.^17-24

Compared to randomized controlled (and “blinded”) studies, these (prospective cohort) studies are less scientifically rigorous; limited to yielding correlations and associations; and more prone to misinterpretations and to mis-attributing to the studied intervention observed reductions in CLABSI rates that are caused by one or more unrecognized and un-controlled confounding factors, some of which are listed in Table 1.

Nevertheless, their design limitations notwithstanding, several of these reviewed studies, in addition typically to not validating the accuracy of their published CLABSI rates, suggest, imply, or conclude (possibly in error) that the studied intervention caused the reduction in the CLABSI rate.^17-24

Table 1 also lists several biases that can cause:

• measured CLABSI rates to under-report the true incidence of infection and
• prospective cohort studies, including those identified during this review, to over-exaggerate the percentage by which an intervention might have reduced the CLABSI rate.^{8-11,21,30,32}

Several characteristics that these prospective cohort studies share in common are discussed in Box D and Table 2 (see: pages 24S₁ and 24S₂, respectively).^17-24 

Discussion

With potentially concerning implications, this review found that the majority of reported CLABSI rates have not been independently validated for accuracy, completeness, and reliability. That many of these published rates may be in error, therefore, is a possibility.

Like report cards that children might write themselves to grade their own school performance without the accuracy of their grades having been confirmed by their teachers,^8-10 published CLABSI rates, with only a few exceptions, are measured, interpreted and reported by hospitals themselves (some as mandated by a state’s laws, others voluntarily) without these rates, both the numerator (i.e., the number of infections) and the denominator (e.g., the number of central-line days) having been independently audited by, for  example, state or federal public-health officials.

Like report cards that children might write themselves to grade their own school performance without the accuracy of their grades having been confirmed by their teachers,^8-10   published CLABSI rates, with only a few exceptions, are measured, interpreted and reported by hospitals themselves (some as mandated by a state’s laws, others voluntarily) without these rates having been independently audited for completeness and accuracy. — Lawrence F Muscarella PhD

Is the ubiquitous use of CLABSI rates to gauge the quality of healthcare sound?

By questioning the validity of the majority of reported CLABSI rates, this review inextricably also questions the soundness of their use by:

1. consumers, among others, to compare the relative safety of hospitals³² (having reasonably, though erroneously, presumed that all reported CLABSI rates have been validated);
2. the CDC, to conclude, based on its state-specific report,⁶ that hospitals are becoming safer;⁷ and
3. Consumer Union, to rate hospitals, label some poor performers, and to urge consumers in its March (2010) issue of Consumer Reports to choose a hospital it labels a top performer.¹²

Questioning the validity of the majority of reported CLABSI rates also raises doubts about the soundness of their use by:

4. private and public insurers, as well as government agencies and federal rules, programs, and policies, to incentivize improved health care by providing to hospitals reimbursements, financial rewards, and other forms of compensation that are conditioned on their reporting (at times, reduced) CLABSI rates (e.g., CMS’s pay-for-reporting program);^{10,24,25-28,33,34}
5. state laws, to reduce the risk of HAIs via greater accountability and transparency;^{1,5,6,8-10,30,32} and
6. clinicians, as a metric to evaluate the effectiveness of interventions implemented in ICUs to reduce CLABSI rates.^17-24,35

Incomplete state laws and federal rules

Similarly questioned is the completeness of state laws that mandate the tracking and reporting of CLABSI rates, and of public and private insurance programs, federal rules, and reimbursement policies that condition financial payments on the reporting of these rates, without requiring that these rates be validated.

Several states currently do not mandate the reporting of CLABSI rates,^6,10 so that an increasing number do demonstrates progress.

But, as long as state laws, among other statutes, programs, rules and policies, do not require that these reported rates be checked to confirm that every CLABSI was counted, in addition to these rates and their comparisons potentially lacking credibility,^8,9,32 the cogency and relevance of these laws may be reasonably questioned.³²

According to one state’s report on HAI initiatives, the validation of infection data, including reported CLABSI rates, “must be considered in any mandatory reporting system to ensure that HAIs are being accurately and completely reported.”³²

Prospective cohort studies

The soundness of the conclusions of several reviewed prospective cohort studies, including one published by the CDC in 2006,³⁵ is similarly questioned.

In general, their conclusions, first, are based on CLABSI data that have not been validated; and, second, suggest or imply that their measured data indicate a causal relationship between the implementation of a studied intervention or bundle of practices and a reduction in the CLABSI rate by a specific percentage.^17-24,35  

While these are most insightful studies, their limiting designs, however, preclude their advancement of such a conclusion, namely, that the studied intervention was responsible for the reduced CLABSI rate (see: Box D and Table 2 on pages 24S₁ and 24S₂, respectively).^17-24

Admittedly, several of these prospective cohort studies acknowledge that their data are limited to yielding correlations and associations. But this limitation is typically not emphasized and, in some instances, is overlooked.^17-24,35

Most notably, these prospective cohort studies are not blinded^20,22 and cannot exclude the possibility that behavioral changes²⁰ or one or more other unrecognized and un-controlled factors—not the studied intervention—caused the observed CLABSI rate reduction.

Moreover, discussed in Table 2 (p. 24S₂), confidence in the suggestions of several of these studies that the evaluated intervention reduced the CLABSI rate is further weakened, because such an observation would require that which these studies typically fail to verify:  that clinicians rigorously adhered to the intervention’s practices.^17-24

Under-reporting, over-exaggerating?

Table 1 lists several biases that, among others, can cause the measured CLABSI rates to under-report the true incidence of infection and to over-exaggerate a studied intervention’s actual effectiveness in ICUs.^{8-11,17-24,32,35}

A possible display of the effects of these biases, significant discrepancies have been identified between the (lower) rates of reported CLABSIs and the (higher) CLABSI rates validated during independent audits (see: Box C).^{6,8-11,30,32,36-38}

Possibly also displaying the effects of these factors, that the national aggregate of CLABSI data listed in the CDC’s state-specific report might, too, under-report the true infection rate cannot be ruled out (see: Box E on p. 24S₁).

Biases

Listed in Table 1, measurement bias resulting from, for example, variations in and reduced sensitivities of the surveillance methods used to measure, interpret, and report CLABSI rates can cause under-reporting of the true CLABSI rate.^10,11,21

Small changes in the sensitivity of these methods, like subtle differences in the subjective interpretation of the CDC’s definition of a CLABSI,³⁰ can cause significant inaccuracies in reported CLABSI rates.

Other biases listed in Table 1 that can similarly cause reported CLABSI rates to under-report the true incidence of infection include financial bias, which may result from, for example, providing hospitals with a financial incentive to report reduced CLABSI rates,^22,25-28,39  and feedback bias.^18-24

Like an open-label drug study, feedback bias can cause behavioral²⁰ changes that affect the measurement of the CLABSI rate (see: Table 2 and Box D).^18-24

Indeed, the designs of several of the reviewed prospective cohort studies ensure that the clinicians are (not blinded and are) told of the intent of the study and of the progress and success of their efforts to reduce the rates of (and costs associated with) CLABSIs in ICUs.^18-24

That reported CLABSI rates may be inaccurate and their ubiquitous use less scientific and meaningful than subjective and conjectural is one of this review’s unexpected and concerning findings.^8,9 

But, while it may be important to reduce CLABSI rates quickly and for as many patients as possible (see: Table 2), such feedback can compromise the study’s scientific integrity and introduce bias that can cause the conclusions of a study evaluating how effectively an intervention might have reduced CLABSI rates in ICUs to be misleading.^{10,11,17-24,35}

Confounding factors

Similarly, confounding factors can introduce confounding bias (see: Table 1), resulting in misinterpretations of the CLABSI data. Unlike randomized controlled studies, prospective cohort studies evaluating the percentage by which an intervention might reduce CLABSI rates can not generally eliminate or control for confounding bias.

Consequently, in addition to being prone to under-reporting the true CLABSI rate, these studies can mis-attribute to the intervention an observed reduction in the CLABSI rate that was actually caused by one or more confounding factors.

Table 1 lists some examples of confounding factors that may vary not just in one ICU but also in a number of different ICUs during the study of an intervention’s effectiveness.^10,11,24

Posing an increased risk of infection?

CLABSI data that have not been validated and under-report the true incidence of CLABSIs can mis-characterize the safety and quality of hospitals and under-estimate the true risk of HAIs in ICUs and in other hospital departments and units, including the GI endoscopy department.

Such misleading infection data could also cause the CDC and other federal agencies to conclude erroneously that health care in the U.S. is becoming safer.^6,7

Most important, as a consequence of such faulty data, ICUs may forgo the implementation of crucial interventions, “miss” important opportunities to prevent HAIs,³² and reallocate limited financial resources and staff hours to other labors appearing (in error) to require more attention, paradoxically posing an increased risk of CLABSIs.

That the publication, use and advancement of inaccurate infection data can, therefore, pose harm to patients stresses the importance of validating reported CLABSI (and other HAI) rates.^8,9,32,38

Conclusions, recommendations

Although increasingly used as a metric to evaluate, rate, and compare the safety and quality of hospitals,^6,8,9,12   the majority of reported CLABSI rates have not been independently validated for accuracy, completeness, and reliability.

That these reported CLABSI rates, therefore, may be inaccurate, not credible,^32,40 pose a risk of harm to patients, and their ubiquitous use less scientific and meaningful than subjective and conjectural is a concerning finding.^8,9 

A number of factors, some of which are listed in Table 1, can affect reported CLABSI rates, causing them to under-report the true incidence of infection and, if these rates are associated with a studied intervention, to over-exaggerate its clinical effect on CLABSI rates in ICUs.^{8-11,21,30,32}

The cautious use of reported CLABSI rates is, therefore, advised, whether published in a CDC report or consumer magazine, or used by, among others, consumers, a state law, governmental agency, federal rule, or health insurer.

In closing, the validation of reported CLABSI rates is recommended and “essential,”³² to ensure that they are accurate, complete, and can be used soundly and without hesitation.

It is also recommended that state laws—as well as public and private insurance and reimbursement programs, rules, and policies that condition financial rewards, payments, or bonuses on the reporting of CLABSI data—be updated and completed to require the validation of reported CLABSI rates.

Whether both the CDC’s state-specific report on CLABSI data and Consumers Union’s article about CLABSI rates in its Consumer Reports’ March (2010) issue will be updated to emphasize more clearly that the majority of their listed CLABSI data have not been independently validated and that, because these data may therefore be inaccurate, their use to compare the safety of hospitals may be unsound is  unclear, though such clarification is encouraged, if not urged.

Finally, the standardization of surveillance methods used to detect and interpret CLABSI rates is also encouraged, and the importance of statistically adjusting these rates for risk factors to account for differences in patient populations (e.g., “patient-mix”⁶) is noted, so that CLABSI rates, their reporting, and their comparisons are more scientifically sound.^10,11

Performing randomized controlled studies to evaluate how effectively an intervention might reduce CLABSI rates in ICUs is recommended, as is also the clearer disclosure by prospective cohort studies that their designs restrict their observations to associations and correlations.

Article by: Lawrence F Muscarella, PhD. Posted: December 9, 2013. LFM Healthcare Solutions, LLC Copyright 2016. LFM Healthcare Solutions, LLC.  All rights reserved. 0002

Lawrence F Muscarella PhD is the owner of LFM Healthcare Solutions, LLC, a Pennsylvania-based quality improvement and consulting company that provides safety services for hospitals, manufacturers and the public. Email Dr. Muscarella for more details.