5
$\begingroup$

I noticed when calculating the variance of the slope and intercept estimators in simple linear regression that the final formulae do not contain any $y_i$'s. While I understand the calculation, I was wondering if there is a simple intuitive argument for this. I assume that the errors being i.i.d. with a fixed variance regardless of $x$ or $y$ is key, but I think there might be something a bit deeper here that I am missing. It just seems like there must be a calculation free, logical, explanation of this.

\begin{equation} y_i = \theta_0 + \theta_1x_i + \epsilon_i, \end{equation}

\begin{equation} Var{[\hat{\theta_0}]}=\frac{\sigma^2\sum\limits_{i=1}^{n}x_i^2}{n\sum\limits_{i=1}^{n}x_i^2-\left(\sum\limits_{i=1}^{n}x_i\right)^2} \end{equation}

\begin{equation} Var{[\hat{\theta_1}]}=\frac{n\sigma^2}{n\sum\limits_{i=1}^{n}x_i^2-\left(\sum\limits_{i=1}^{n}x_i\right)^2} \end{equation}

Improve this question
$\endgroup$
3
  • $\begingroup$ Those formulas on the denominator should normally be avoided, since they're numerically unstable -- with the possible exception of hand calculation (provided you understand when you're losing accuracy using them and take appropriate steps). $\endgroup$ Commented Aug 29, 2017 at 9:55
  • $\begingroup$ @Glen_b can you provide a reference, or explanation so I can further explore this point. $\endgroup$ Commented Aug 29, 2017 at 11:08
  • $\begingroup$ Up to a scale-factor this is just the variance of the $x$'s, on which see Wikipedia's Algorithms for calculating variance: Naive algorithm, particularly the mentions of cancellation and catastrophic cancellation (see also the article on Loss of significance). Fast, stable one-pass algorithms have been around at least since Welford (1962); the issue is addressed in a number of posts here; it's a problem when the mean is very large compared to s.d. $\endgroup$ Commented Aug 30, 2017 at 0:47

1 Answer 1

Reset to default
2
$\begingroup$

Well the $y_i$'s are present in a way in your $\sigma^2$: the larger the spread of the $y_i$'s around $\theta_0 + \theta_1 x_i$, the larger the variances of these estimators.

The clue is that these equations are only correct under the model assumption of constant variance $\sigma^2$. Once you set the $x_i$'s and assume constant variances, the $y_i$'s are known to be $Y_i \sim N(E(Y_i|x_i),\sigma^2)$ and thus no longer "affect" the theoretical variances of the estimators.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ The $y$'s are present in the estimate of $\sigma^2$. $\endgroup$ Commented Aug 29, 2017 at 9:53
  • $\begingroup$ @Glen_b's comment is a really important point, the y's are still present in RSS and hence the estimate $\sigma^2$. $\endgroup$ Commented Aug 29, 2017 at 11:11

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.