How do you measure whether a quality measure is reliable? What does the term “reliable” even mean?
In the world of quality measurement, reliability is the ability to confidently tell the difference between a high-performing individual (or physician or hospital) and a low-performing individual. One can think of reliability as a signal-to-noise measure; if the signal is stronger, it is easier to identify high-quality providers, but if the measure has a lot of noise, it will be difficult to distinguish a high quality provider from a low-quality provider.
Today I review how to measure reliability for both continuous and binary quality measures.
The reliability statistic for a given measure is calculated as:
- Reliability = σ2b / [σ2b + (σ2w / n)]
where
- σ2b = the between provider variance
- σ2w = the average within-provider variance
- n = number of observations for a given provider
The framework above is most useful for determining the reliability of a continuous measure. Many quality measures, however, are binary in nature: Did the physician conduct an A1C test? Did the physician give the patient beta blockers after an acute myocardial infarction (AMI)? Did the patient die during the episode of care? These variables take on a value of 1 if the answer is yes and 0 otherwise.
Adams (2009) adapts the framework outlined above for the case of these binary variables. To do this, the author uses a beta-binomial model.
“The beta-binomial model is based on the beta distribution for the ‘true’ physician scores. The beta distribution is a very flexible distribution on the interval from 0 to 1. This distribution can have any mean in the interval and can be skewed left or right or even U-shaped. It is the most common distribution for probabilities on the 0-1 interval. The beta-binomial model assumes the physician’s score is a binomial random variable conditional on the physician’s true value that comes from the beta distribution.”
The beta distribution depends on two parameters: α and β. If these two parameters are known, one can estimate the variance terms in the equation above as:
- σ2b = (αβ) / [(α+β+1)*(α+β)2]
- σ2w = p(1-p)
The value of p is the rate of success of the quality measure (i.e., when the quality measure equals 1). Knowing this, we can calculate reliability for a binary quality score as follows:
- Reliability = σ2b / [σ2b + (p(1-p)/n)]
In practice, however, p, α and β are not known. To estimate p, one can simply use the observed pass rate from the data. To estimate the values of α and β, first consider the relationship between α and β and the mean and variance of the beta distribution.
- μ =α / (α+β)
- σ2 = (αβ) / [(α+β+1)*(α+β)2]
To estimate α and β, the paper recommends using a maximum likelihood estimator. The betabin SAS macro does this in SAS. Alternatively, one can estimate α and β using a methods of moments estimator, as shown in this example.
Source:
- John L. Adams. The Reliability of Provider Profiling: A Tutorial. RAND, 2009.