Unbiased Analysis of Today's Healthcare Issues

Duan’s Smearing Estimator

Written By: Jason Shafrin - Nov• 16•10

The Problem

Many times, researchers wish to transform the dependent variable of a regression in order to estimate parameter values.  Performing the transformation, however, complicates the calculation of the expected value of the dependent variable on the untransformed scale.  Assume, the Yi is the dependent variable. Assume the function g is used to transform the dependent variable as follows:

  • ηi = g(Yi)
  • Yi = h(ηi)
  • h=g-1

The easiest way to image this functions is think of g as the ln function and h as the exp function. In health economics, researchers often use a log transformation to attenuate problems related to a heavily right-skewed distribution. In this case, one would estimate the following regression:

  • ηi = xiβ + εi
  • εi~F(iid), E(εi)=0; Var(εi)=σ2

One can estimate β consistently as follows:

  • β = (X’X)-1X’η

What is the predicted value of the dependent variable? Calculating this is not as easy as it seems:

  • E(Y0) = E[h(x0β + ε)] E[h(x0β)]

For instance, it is well known that using the log transformation, the expected value of the depended variable is equal to: exp(x0β + σ2/2). In cases where we do not know the true distribution of the error term, however, then calculating the expected value of the error term is more difficult. The solution is Duan’s Smearing Estimate.

The Solution

The goal is to estimate the following:

  • E(Y0) = E[h(x0β + ε)] = ∫ h(x0β)dF(ε);

Duan states that: “Without knowing the error distribution function F or a reliable parametric form for it, we estimate F by the empirical cdf of the estimated residuals

  • Fn(e) = n-1Σi=1 to n [I{εi ≤ e}]

where εi = ηi – xiβ denotes the residual, I{⋅} denotes the indicator function of the event “⋅”.

By substituting the coefficients from the original regression, we can estimate the expected value of the untransformed dependent variable as:

  • Ê(Y0) = ∫ h(x0β)Fn(ε)
  • Ê(Y0) = n-1Σi=1 to n [h(x0β + εi)]

To see how the smearing estimator works in practice see this example.  I examine (made-up) figures of how Medicare spending varies by age.  One can see that the average spending level is $6516.  Retransforming the dependent variable on the log scale and using the naive estimate produces an estimate expected value of $4,853.  Using the Duan smearing estimator, however, we get much closer to the actual spending level.  The estimated average spending level is $6,725, much closer to the actual figure than the naive estimate.

This example assumes the function g() is the natural log function.  You can see more detail on dealing with log estimation for homoskedastic and heteroskedastic erros here.


You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

One Comment

  1. Steve Samuels says:

    The expected value of the depended variable is equal to: exp(x_0β + σ^2/2) is true only when Y is normally distributed.

Leave a Reply

Your email address will not be published. Required fields are marked *