1
$\begingroup$

If I have a $(X, Y)$ dataset and want to model $y = f(x, \beta)$. In that case for OLS, I would have $$e(x_i, \beta) = f(x_i, \beta) - y_i$$ Then obviously I would have $$SSE = \sum_{i}e(x_i, \beta)^2$$ and in a closed form $$\frac{\partial SSE}{\partial \beta}|_{\hat{\beta}} = 0$$ while in a numeric form, $$\underset{\beta}{\text{min}} SSE(\beta)$$

But according to the wikipedia article, this is only the case when the variance is absolutely constant.

Now, what does this statement mean? Does it mean that $|e(x_i, \hat{\beta})| = const..., \forall x_i$, i.e. $|e(x_1, \hat{\beta})| = |e(x_2, \hat{\beta})| = |e(x_3, \hat{\beta})| = |e(x_4, \hat{\beta})| = .....\text{so on and so forth} $

Now, failing this assumtion, a better estimator of $\beta$ would be $$\underset{\beta}{\text{min}} \quad \sum_{i} w_i e(x_i, \beta)^2$$ Where $w_i = 1/{\sigma_i}^2$

How exactly is $\sigma_i$ defined here? Is it $\sigma_i = \sqrt{e(x_i, \beta)^2}$? How is the equality of $\sigma_i$ enforced?

$\endgroup$
6
  • $\begingroup$ Do you have any apriori knowledge on the weights? $\endgroup$ Commented May 21, 2024 at 12:08
  • $\begingroup$ Not really. All I have apriori is the $(X, Y)$ dataset and the generic model $y = f(x, \beta)$ to which I would want to fit my dataset in the process of estimating $\hat{\beta}$ $\endgroup$ Commented May 21, 2024 at 12:17
  • $\begingroup$ Is there strong evidence that there is heteroskedasticity? $\endgroup$ Commented May 21, 2024 at 12:20
  • $\begingroup$ Wouldn't call it strong, but $|e(x, \hat{\beta})|$ does not necessarily show a monotonic relationship with $x$. Of course there are select areas where $|e(x, \hat{\beta})|$ are larger where the model $f(x, \beta)$ changes signs especially in its higher order derivatives, other than that I don't think so. That's why I wanted to know, what would be a forebearing sign of heteroskedsticity and what does constant variance even mean mathematically. $\endgroup$ Commented May 21, 2024 at 12:27
  • 1
    $\begingroup$ If could say, that there is some way you could think of variance clustering, then weighted least squares could be useful, otherwise not. $\endgroup$ Commented May 21, 2024 at 12:40

2 Answers 2

Reset to default
1
$\begingroup$

Since the article you linked is focussed on linear regression, I will answer assuming that $f(x_{i},\beta) = x_{i}^{T}\beta$, for simplicity.

The simplest model, under homoskedasticity is assumed to be $y_{i} = x_{i}^{T}\beta + \varepsilon_{i}, i = 1, 2, \dots, n$, with $E[\varepsilon_{i}] = 0$ and $E[\varepsilon_{i}^{2}] = \sigma^{2}$ (if you want you can make these expectations conditional on $X$, but for simplicity I will assume that regressors $X$ are fixed). In matrix notation this would be $Y = X\beta + \varepsilon$ with $E[\varepsilon] = 0$ and $E[\varepsilon\varepsilon^{T}] = \Sigma$ with $\Sigma = \sigma^{2}I$, with $I$ an identity matrix of size $n$. The variance being constant refers to the fact that the true residuals $\varepsilon_{i}$ all have variance $\sigma^{2}$ under this model. If this is the case, then OLS, which minimises $$ SSE(\beta) = \sum_{i=1}^{n}(y_{i} - x_{i}^{T}\beta)^{2} = (Y-X\beta)^{T}(Y-X\beta), $$ is optimal among the class of linear unbiased estimators (BLUE), in the sense that $var(\hat{\beta})$ is smallest among this class.

Suppose now that you have heteroskedasticity, then the model is the same as in th above, except that now it is assumed that $E[\varepsilon\varepsilon^{T}] = \mathrm{diag}(\sigma_{1}^{2}, \sigma^{2}_{2}, \dots, \sigma_{n}^{2})$, so each residual $\varepsilon_{i}$ has a different variance, but all residuals are still uncorrelated. In this case, OLS is no longer optimal, and the weighted least squares estimator is now BLUE.

In-sample you will of course never find that things match up perfectly with your model assumptions, i.e. under homoskedasticity even you won't find that all residuals $e^{2}(x_{i},\beta)$ are exactly the same, but you can do tests to see whether this is the case (see e.g. here). Similar tests for autocorrelation (i.e. fully unrestricted covariance matrix $\Sigma$) also exist, but then you should use generalized least squares.

Since in practice you never know the true $\sigma_{i}^{2}$ under heteroskedasticity, there is a procedure called feasible weighted/generalized least squares (see here), which aims to estimate the weights.

$\endgroup$
1
$\begingroup$

The $\sigma_i$ are the standard uncertainties in the experimental measurements: that is to say, for a fixed "true" value of $y_i$, there exists a probability density distribution over the measured value of $y_i$ that is obtained, and $\sigma_i$ is the standard deviation of that probability density distribution. See the BIPM documents here for lots and lots of detail.

When the Wikipedia article talks about the special case where "the variance is absolutely constant", the variance it's talking about is $\sigma_i^2$, and "absolutely constant" means the same for every $i$ (and not necessarily independent of $\hat{\beta}$, for reasons set out by Orear). This equality of all the $\sigma_i$ can happen if the uncertainty values come from what the BIPM documentation calls "a type B estimation of the standard uncertainty", i.e. from the experimenter knowing, before carrying out the measurements, about the characteristics of the measuring instrument and being able to provide an estimate of the uncertainty from that knowledge - if it's the same measuring instrument each time, you may get the same uncertainty estimate each time that way.

$\endgroup$

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.