2
$\begingroup$

Consider the following location model.

$$ x_i = \mu + u_i, (i = 1,\dots, n), $$

where $u_i$ are $i.i.d.$ with density function $f_0$. Hence, $x_i$ are $i.i.d.$ with density function $f_0(x-\mu)$. It usually is of interest to find the estimator of $\mu$. One example is the Maximum-Likelihood-Estimator for location $\mu$ (a special case of $M$-estimators) defined to be

$$ \widehat\mu(x_1, \dots, x_n) := \arg\min_\mu \sum_{i=1}^n -\log f_0(x_i-\mu). $$

I read that this estimator is NOT scale equivariant. That is, for a constant $c \in \mathbb R$,

$$ \widehat\mu(cx_1, \dots, cx_n) \neq c\widehat\mu(x_1, \dots, x_n). (*) $$

I do not understand why this is the case. I suppose whether $(*)$ holds depends on the function $f_0$, doesn't it? Could anyone explain this to me, please? Some examples are appreciated. Thank you!

UPDATE: As an example, let us consider the normal distribution $N(\mu, \sigma^2)$. The MLE for $\mu$ (regardless of $\sigma$) can be shown to be

$$ \widehat\mu(x_1, \dots, x_n) = \frac{\sum_{i=1}^n x_i}{n}. $$

Moreover, if you multiply $x_i$ by $c$, it is also straightforward to show that

$$ \widehat\mu(cx_1, \dots, cx_n) = \frac{c\sum_{i=1}^n x_i}{n} = c\widehat\mu(x_1, \dots, x_n). $$

Also note in all these derivation one did not use $\sigma$ at all. Hence, I do NOT see why this estimator is not scale equivariant. Could anyone give me an example where the identity

$$ \widehat\mu(cx_1, \dots, cx_n) = c\widehat\mu(x_1, \dots, x_n). $$

does NOT hold, please? Thank you!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Because you need it to be a location and scale family, $f((x - \mu)/\sigma)$, and the estimation method needs to include a data-based estimator of $\sigma$ as well, e.g., the MAD (median absolute deviation of the residuals). For most but the normal case, the scale parameter makes a difference in the log of the PDF.

Addendum

Now that I'm not distracted by the NCAA basketball finals, I can add some info, hopefully with fewer mistakes...

The next step (for sufficiently smooth densities) in the question's second displayed equation (with $\arg\min$) is to differentiate and set to zero -- i.e., solve the equation $$ \sum \psi(x_i - \hat\mu) = 0 $$ where $\psi(z) = -\frac d{dz}\log f_0(z)$. If $f_0(z) = \frac1{\sqrt{2\pi}\sigma}\exp\{-\frac12z^2/\sigma^2\}$, we have $-\log f_0(z) = \frac12\log(2\pi) + \log\sigma + \frac12z^2/\sigma^2$, so that $\psi(z) = z/\sigma^2$ -- and thus we need to solve the equation $\frac1{\sigma^2}\sum(x_i-\hat\mu) = 0$. Since $\sigma\ne0$, we can just canel it out and it plays no further role in solving the equation. Thus, $\hat\mu$ is scale equivariant.

However, suppose that $f_0(z) = \frac1{\pi\sigma}\cdot\frac1{1 + z^2/\sigma^2}$ is the Cauchy density with $\sigma$ as a scale parameter. We now have $-\log f_0(z) = \log{\pi\sigma} + \log(1+z^2/\sigma^2)$, so that $\psi(z) = \frac{2z/\sigma^2}{1+z^2/\sigma^2} = \frac{2z}{\sigma^2+z^2}$. In solving $\sum\psi(x_i-\hat\mu)=0$, $\sigma$ does not cancel out, and its value affects the solution! That is, with this Cauchy density as the basis for the M estimation, $\hat\mu$ is not scale equivariant.

The gist of my original answer is that you should include the scale parameter in the equation: $$ \sum \psi\{(x_i - \hat\mu)/\hat\sigma\} = 0 $$ The classic M-estimation example uses Huber's $\psi(z) = \mbox{sign}(z)\cdot\min(|z|,c)$ where $c$ is a suitable tuning constant, e.g. 1.34. A popular scale estimator is $\hat\sigma = \mbox{med}|x_i - \mbox{med}_j\{x_j\}|$, then obtain $\hat\mu$ as an iteratively weighted mean with weights $w_i = \frac{\psi\{(x_i - \hat\mu)/\hat\sigma\}}{(x_i - \hat\mu)/\hat\sigma}$ (and with, say, the median as the initial $\hat\mu$. This estimator is scale equivariant because $\hat\sigma$ is scaled proportionally to the scale of the data.

Improve this answer
$\endgroup$
9
  • $\begingroup$ Thanks. However, I still do not get what you are saying (see my update). Could you give an specific example to illustrate your point, please? $\endgroup$ Commented Apr 7, 2015 at 1:17
  • $\begingroup$ Here are some examples for you to try. Let f(x) be the normal density and g(x) be the Cauchy density. Look at graphs of log f(x), log f(2x), log g(x), and log g(2x). Note that the first two are identical except for an additive constant, hence minimizing the sum of several such things will give the same weights to each term regardless of scale. But that is not true for the Cauchy ones. $\endgroup$ Commented Apr 7, 2015 at 1:51
  • $\begingroup$ In the normal case, we have $\log f(x) = -\log\sqrt{2\pi} - \log\sigma -\frac{(x-\mu)^2}{2\sigma^2}$ and $\log f(2x) = -\log\sqrt{2\pi} - \log\sigma -\frac{(2x-\mu)^2}{2\sigma^2}$. Unlike what you suggested above, the difference between the two are not an additive constant. Sorry, but I still do not see your point. $\endgroup$ Commented Apr 7, 2015 at 2:11
  • $\begingroup$ Just take mu equal to zero and it'll be clearer. Or if you want nonzero mean, you need to put in 2mu also, since you're asking about EQUIvariance not INvariance. $\endgroup$ Commented Apr 7, 2015 at 2:17
  • $\begingroup$ Take $\mu=0$ we have $\log f(x) = -\log\sqrt{2\pi} - \log\sigma - \frac{x^2}{2\sigma^2}$ and $\log f(2x) = -\log\sqrt{2\pi} - \log\sigma - \frac{2x^2}{\sigma^2}$. Still the difference is NOT additive. Moreover, we are trying to estimate $\mu$. How can you take it to be 0, please? Sorry if my stupidity offends you. $\endgroup$ Commented Apr 7, 2015 at 2:22

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.