Duan’s Smearing Estimator

The Problem

Many times, researchers wish to transform the dependent variable of a regression in order to estimate parameter values.  Performing the transformation, however, complicates the calculation of the expected value of the dependent variable on the untransformed scale.  Assume, the Y_i is the dependent variable. Assume the function g is used to transform the dependent variable as follows:

• η_i = g(Y_i)
• Y_i = h(η_i)
• h=g^-1

The easiest way to image this functions is think of g as the ln function and h as the exp function. In health economics, researchers often use a log transformation to attenuate problems related to a heavily right-skewed distribution. In this case, one would estimate the following regression:

• η_i = x_iβ + ε_i
• ε_i~F(iid), E(ε_i)=0; Var(ε_i)=σ²

One can estimate β consistently as follows:

• β = (X’X)^-1X’η

What is the predicted value of the dependent variable? Calculating this is not as easy as it seems:

• E(Y₀) = E[h(x₀β + ε)] ≠ E[h(x₀β)]

For instance, it is well known that using the log transformation, the expected value of the depended variable is equal to: exp(x₀β + σ²/2). In cases where we do not know the true distribution of the error term, however, then calculating the expected value of the error term is more difficult. The solution is Duan’s Smearing Estimate.

The Solution

The goal is to estimate the following:

• E(Y₀) = E[h(x₀β + ε)] = ∫ h(x₀β)dF(ε);

Duan states that: “Without knowing the error distribution function F or a reliable parametric form for it, we estimate F by the empirical cdf of the estimated residuals

• F_n(e) = n^-1Σ_{i=1 to n} [I{ε_i ≤ e}]

where ε_i = η_{i –} x_iβ denotes the residual, I{⋅} denotes the indicator function of the event “⋅”.

By substituting the coefficients from the original regression, we can estimate the expected value of the untransformed dependent variable as:

• Ê(Y₀) = ∫ h(x₀β)F_n(ε)
• Ê(Y₀) = n^-1Σ_{i=1 to n} [h(x₀β + ε_i)]

To see how the smearing estimator works in practice see this example.  I examine (made-up) figures of how Medicare spending varies by age.  One can see that the average spending level is $6516.  Retransforming the dependent variable on the log scale and using the naive estimate produces an estimate expected value of $4,853.  Using the Duan smearing estimator, however, we get much closer to the actual spending level.  The estimated average spending level is $6,725, much closer to the actual figure than the naive estimate.

This example assumes the function g() is the natural log function.  You can see more detail on dealing with log estimation for homoskedastic and heteroskedastic erros here.

Source:

• Naihua Duan  Smearing Estimate: A Nonparametric Retransformation Method, Journal of the American Statistical Association, Vol., 78, No. 3838. (Sep., 1983), pp. 605-610.