9
$\begingroup$

In ridge regression with design matrix $X$, outcomes $y$, fixed regularization parameter $\lambda$, and errors $\epsilon\sim\mathcal{N}(0, \sigma^2I)$, the computations for the ridge regression coefficients $\hat\beta$ (aka the solution to $\arg\min_b \big[(y-Xb)'(y-Xb) + \lambda b'b\big]$) and their variance-covariance matrix $var(\hat\beta)$ are:

$$ \begin{align*} M &:= (X'X + \lambda I)^{-1}X' \\ \hat\beta &= My \\ var(\hat\beta) &= \sigma^2MM' \end{align*} $$

The computation on $var(\hat\beta)$ relies on $\lambda$ being treated as a constant value, which makes $M$ constant. Thus, we can apply the identity $var(My) = Mvar(y)M'$ and the fact that $var(y) = \sigma^2I$ to derive $var(\hat\beta)$.

However when $\lambda$ is selected using $X$ and $y$ (for my application via cross-validation), $\lambda$ and $M$ become stochastic. How can $var(\hat\beta)$ be updated to deal with a stochastic $\lambda$ selected via cross-validation?

$\endgroup$

1 Answer 1

2
$\begingroup$

I'm guessing that these equations are maximum likelihood solutions. The MLE of a parameter takes as its variance-covariance matrix the inverse of the second derivative of the likelihood function. What this means is that var$(\hat B)\sim∂^2L($data, $\hat B)/∂B^2$. If you include the lambda parameter, this partial derivative does not change. You introduce a covariance between B and lambda. The fact that you're optimizing lambda via cross-validation rather than MLE doesn't change the story for beta.

$\endgroup$
1
  • $\begingroup$ I've updated my question to clarify how $\hat\beta$ and $var(\hat\beta)$ are derived -- $\hat\beta$ is selected to minimize the sum of squared residuals plus a penalty term, and $var(\hat\beta)$ is based on a simple identity that relies on constant $\lambda$ values. I don't understand link to MLE that you're drawing in this answer -- could you clarify? $\endgroup$ Commented Jul 9, 2014 at 0:50

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.