I understand that the trace of the projection matrix (also known as the "hat" matrix) X*Inv(X'X)*X' in linear regression is equal to the rank of X. How can we prove that from first principles, i.e. without simply asserting that the trace of a projection matrix always equals its rank?
I am aware of the post Proving: "The trace of an idempotent matrix equals the rank of the matrix", but need an integrated proof.
If $X$ is $n \times m$ with $m \le n$ and has full rank, then $rank (X) = \min(n,m) = m$, and we know $(X^T X)^{-1}$ exists.
By commutativity of the trace operator, we have
$$tr(H) := tr (X (X^T X)^{-1} X^T) = tr (X^T X (X^T X)^{-1} ) = tr[I_m] = m$$
Let $X$ be an $n \times r$ matrix with full rank, i.e. $\text{rank}X = r$ (this is necessary for $(X^TX)^{-1}$ to exist).
Then, using the identity $\text{tr}(AB) = \text{tr}(BA)$, we have
$\text{tr}[X(X^TX)^{-1}X^T] = \text{tr}[X^TX(X^TX)^{-1}] = \text{tr}[I_{r \times r}] = r = \text{rank}X$, as desired.
The hat matrix is a projection matrix so it's idempotent. i.e. it only has eigenvalues of 0 or 1.
The eigenvalues of $A^n$ are $\lambda^n$, where $\lambda$'s are the eigenvalues of $A$, which holds for any square matrix $A$.
Since H is idempotent: $H=H^2=...=H^n=...$, therefore $\lambda = \lambda^n, n=1,2,...$. And since X is full column ranked, H is full ranked, thus the number of 1 as its eigenvalues is just rank(H). Since the trace is also the sum of the eigenvalues and the sum of eigenvalues is just rank(H). Therefore the tr(H) = rank(H) = colrank(X) = rank(X). $\lambda$.
