Unbiased Analysis of Today's Healthcare Issues

Are quality bonus payments based on hospital readmissions reliable?

Written By: Jason Shafrin - Nov• 15•16

Maybe not.  That is the answer from a study by Thompson et al. (2016).  Using data from the Healthcare Cost and Utilization Project (HCUP) State Inpatient Databases (SID) for six states (AR, FL, IA, MA, NY,WA)  from 2011 to 2013, the authors measure hospital performance reliability for the Hospital Readmission Reduction Program (HRRP).  The define reliability as follows:

Measure reliability is determined by the between-hospital variation in event rates (the signal ) and the within-hospital variation in event rates (the noise ) (Adams 2009; Adams et al. 2010). Reliability (R ) is then estimated as R = signal /(signal + noise ), and it ranges from zero to one, with reliability increasing as this ratio approaches one.

Is reliability a big issue?  Isn’t this just random noise that gets evened out over time?  Statistically the answer is yes, but if reimbursement based on readmissions is tied mostly to random chance rather than quality of care, there is little point in tying reimbursement to readmissions.  Of course, smaller hospitals are less likely to have reliable measures due to a smaller sample size of admissions.

The conditions evaluated by the study were acute myocardial infaction (AMI), congestive heart failure (CHF), and pneumonia (PN); more recently, it has expanded to include readmissions for chronic obstructive pulmonary disease (COPD), total hip and/or knee arthroplasty (THKA), and coronary artery bypass graft surgery (CABG).  For each of these conditions, the authors followed CMS’ approach for measuring readmission rates:

…we used hierarchical logistic regression models to estimate the predicted to expected number of readmissions for each hospital and condition, given their hospital case mix (i.e., the P/E ratio or excess readmission ratio); this ratio was then multiplied by the observed readmission rate for the entire condition-specific cohort. All models were adjusted for the comorbidities used in the CMS risk-adjustment models.

Reliability was measured as the ratio of the between-hospital standard devaiation in average readmission rates divided by the sum of the between-hospital standard deviation in average readmission rates and the within-hospital standard deviation in readmission rates.  The within-hospital variance will vary across hospitals due to both sample size and the proportion of readmissions (which affects the variance as readmissions follow a binomial distribution).

Using a benchmark reliaability of 0.70, the authors found that HRRP reliability was mixed:

  • AMI (R = 0.58), 18.6% of hospitals exceed R>0.70
  • CHF (R = 0.61), 16.6% of hospitals exceed R>0.70
  •  COPD (R = 0.65), 28.1% of hospitals exceed R>0.70
  • PN (R = 0.68), 40.2% of hospitals exceed R>0.70
  • CABG (R = 0.78), 86.6% of hospitals exceed R>0.70
  • THKA (R = 0.94), 100% of hospitals exceed R>0.70

Overall, the authors found that readmission measures for surgical admissions are more reliable than those for medical admissions.  They conclude the following:

For measures to be reliable for group-level comparisons, they must have sufficient between-group variation (e.g., physicians, hospitals, accountable care organizations) and adequate sample size. The absence of either of these elements limits the measure s usefulness in hospital performance profiling. For instance, the condition with the largest sample size, CHF, has among the worst reliability because hospital-level RSRRs do not vary substantially (range: 16.1 36.2 percent). Conversely, RSRRs for CABG have much higher reliability, despite its small sample size, because the variation in RSRRs is much larger (range: 11.4 47.1 percent).

A very interesting study that won a Best of AcademyHealth Annual Research Meeting Award for 2016.


You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Leave a Reply

Your email address will not be published. Required fields are marked *