Measuring Hospital Quality

How does one measure hospital quality? Quality occurs along multiple dimensions. Thus, to summarize overall quality, one must create a weighting scheme to compete the distinct quality measures in a single measure. In most cases, quality measures should also account for differences in patient case mix. Hospitals should not be punished with lower quality scores for treating sicker patients. Various risk adjustment methodologies can be applied to account for variation in patient case mix across hospitals.

Today, I review one proposed method for creating composite mortality measures to predict hospital mortality rates after surgery.

Methodology

Step 1: Calculate Risk Adjusted Rates for Each Measure

• Y_i = O_i/E_i,
• O_i = n_i^-1Σ_jy_ij
• E_i = n_i^-1Σ_jp_ij

where p_ij is the predicted probability based on a logit function [i.e., p_ij=P(y_ij=1|X_ij)] and X_ij are patient covariates such as age, gender, race, and health status.  Y_i is a Kx1 vector of risk-adjusted mortality rate for hospital i.

Step 2: Calculate Variance-Covariance Matrix

At the first level, the distribution of the mortality estimate conditional on the structural parameter is:
E(Y_i | μ_i) = μ_i , and Var(Y_i | μ_i) = V_i

where Y_i is defined as above and μ_i is the corresponding underlying structural quality parameter that represents the average mortality rate that a typical patient could expect at this hospital, and V_i is the KxK sampling variance for the estimates in Y_i.

Step 3: Conduct Hospital-Level Analysis
Other factors such as hospital-specific factors such as hospital volume, teaching status, and other factors can also affect expected mortality rates. These variables do not vary by patient; only across hospital.  However, one can explicitly incorporate these into a composite measures using the following framework:

• E(μ_i) = Z_iβ
• Var(μ_i) = Ω

where β is a KxJ matrix of coefficients capturing the effect of hospital characteristic j on patient outcome k and Z_i is a Jx1 vector of observable characteristics of hospital i that are thought to be related to patient outcomes. Ω is the variance-covariance matrix in the hospital-level quality parameters summarizing the relationships across different dimensions of hospital quality.

One can estimate β using a weighted least squares regression of Y on Z, where one weights by n_i, the number of patients in hospital i.

One can estimate Ω as follows:

• Ω=N^-1 Σ {(Y_i) – Z_iβ)'(Y_i) – Z_iβ) – V_i}
• Ω=Var(Y_i) – mean(V_i)

Here, V_i is the mean sampling-error covariance matrix. The i-j element of Ω is weighted by the product of n_i and n_j. In certain cases, the variance covariance matrix may not be positive-semi definite (e.g., correlations are greater than 1). If this occurs, one can replace the corelation matrix with the nearest positive-semi-definite correlation matrix.

Step 4: Bayesian Composite Score
The estimates Y_i are a noisy measures of hospital quality. Further, on must determine how much each measure should contribute to the overall score. Using an empirical Bayesian approach, the weights depend on both the signal (&Sigma) and noise (V_i) variances. Thus, the final hospital score is:

• m_i = Y_iW_i + Z_iβ(I-W_i)

where I is a KxK identity matrix and W_i is a KxK weighting matrix estimated by:

• W_i = (Ω+V_i)^-1Ω

In other words, the weight is the ratio of the signal variance to the total variance. Thus, estimated composite score places more weight on a hospital’s own outcome (Y_i) when the signal is high, but shrinks back toward a conditional mean (Z_iβ) when the signal ratio is low.

Attractive Features

1. The composite measure incorporate information in a systematic way from many quality measures into a single outcome.
2. The parameters are consistently estimated as the number of hospitals increases.
3. The estimates maintain many aspects of the existing Bayesian approaches while simplifying the complexity of the estimation.

Drawbacks

There are some drawbacks to this approach. For instance, this method does not incorporate subjective weights. In the example, a hospital has multiple mortality measures. Since the there is no a priori reason to weight one mortality measure more than another, this approach is useful. However, hospitals may be evaluated based on process of care, patient outcome, and patient satisfaction measures simultaneously. One may wish to place more importance on certain measures than others. However, the approach described above may be useful for calculating domain-level composite scores and then one can aggregate the domain level composites using subjective weights.

Source

• Dimick, J. B., Staiger, D. O., Osborne, N. H., Nicholas, L. H. and Birkmeyer, J. D. (2012), Composite Measures for Rating Hospital Quality with Major Surgery. Health Services Research, 47: 1861–1879. doi: 10.1111/j.1475-6773.2012.01407.x
• Justin B. Dimick, Douglas O. Staiger, Onur Baser, Zhaohui Fan, John D. Birkmeyer. Composite Measures for Predicting Hospital Mortality with Surgery. Working Paper, 2008.