5
$\begingroup$

I am reading in the fabulous book of "Exploratory Multivariate Analysis by Example Using R" 2nd edition by Husson, however when I came across this sentence about PCA loadings and their calculation I couldn't get its math or how to prove it in R code:

Loadings are interpreted as the coefficients of the linear combination of the initial variables from which the principal components are constructed. From a numerical point of view, the loadings are equal to the coordinates of the variables divided by the square root of the eigenvalue associated with the component.

How can loadings be calculated given the above statement in this R example from the variables divided by the square root of the eigenvalue of the principal component?

I know that each principal component is a linear combination of the variables and loadings are the coefficients of these linear combinations.
Example

A <- as.matrix(data.frame(mtcars[,c(1:7,10,11)]), nrow = 9, byrow = TRUE) S <- scale(A) pca_svd <- svd(S) pca_svd$v # here is the loading matrix [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [1,] -0.393 0.0275 -0.2212 -0.00613 -0.321 0.7202 -0.3814 -0.1247 0.1149 [2,] 0.403 0.0157 -0.2523 0.04070 0.117 0.2243 -0.1589 0.8103 0.1627 [3,] 0.397 -0.0889 -0.0783 0.33949 -0.487 -0.0197 -0.1823 -0.0642 -0.6619 [4,] 0.367 0.2694 -0.0172 0.06830 -0.295 0.3539 0.6962 -0.1657 0.2518 [5,] -0.312 0.3417 0.1500 0.84566 0.162 -0.0154 0.0477 0.1351 0.0381 [6,] 0.373 -0.1719 0.4537 0.19126 -0.187 -0.0838 -0.4278 -0.1984 0.5692 [7,] -0.224 -0.4840 0.6281 -0.03033 -0.148 0.2575 0.2762 0.3561 -0.1687 [8,] -0.209 0.5508 0.2066 -0.28238 -0.562 -0.3230 -0.0856 0.3164 0.0472 [9,] 0.245 0.4843 0.4641 -0.21449 0.400 0.3571 -0.2060 -0.1083 -0.3205 pca_svd$d # here are the eigenvalues [1] 13.241 8.034 3.954 2.866 2.383 1.959 1.805 1.347 0.829 sqrt(pca_svd$d) # the square root of the eigenvalues [1] 3.639 2.834 1.988 1.693 1.544 1.400 1.343 1.161 0.911

So the A matrix has 32 rows and 9 columns (variables), so what is meant by variable coordinates and what does this statement really mean?

Update: using FactoMineR package

When I use the FactoMineR package which the above book deals with, I even get more confused as the meaning of the statement in question, see the code below:

library(FactoMineR) res.pca <- FactoMineR::PCA(mtcars[, c(1:11)], ncp = 9, quali.sup = c(8, 9)) head(res.pca$var$coord) # here store are the coordinates of the variables R> head(res.pca$var$coord) Dim.1 Dim.2 Dim.3 Dim.4 Dim.5 Dim.6 Dim.7 Dim.8 Dim.9 mpg -0.935 0.0397 -0.1571 -0.00315 0.1373 0.25338 0.1236 -0.0302 0.01712 cyl 0.957 0.0227 -0.1792 0.02095 -0.0501 0.07893 0.0515 0.1960 0.02423 disp 0.945 -0.1283 -0.0556 0.17477 0.2083 -0.00692 0.0591 -0.0155 -0.09860 hp 0.873 0.3888 -0.0122 0.03516 0.1261 0.12453 -0.2257 -0.0401 0.03751 drat -0.742 0.4930 0.1065 0.43535 -0.0693 -0.00541 -0.0155 0.0327 0.00567 wt 0.888 -0.2481 0.3222 0.09846 0.0802 -0.02947 0.1387 -0.0480 0.08479 # actually these are the loadings (V . Sigma) as proof to that: res.pca$svd$V %*% diag(res.pca$svd$vs) == res.pca$var$coord # TRUE

So how can we compute loadings according to the statement in question of FactoMineR book and package from the variable coordinates when the coordinates themselves are actually the loadings matrix as we know it ($V \cdot \Sigma$)?

Accordingly, my guess is that this statement could read like the following:

Loadings are interpreted as the coefficients of the linear combination of the initial variables from which the principal components are constructed. From a numerical point of view, the loadings are equal to the coordinates of the variables divided which are the eigenvectors scaled up by the square root of the eigenvalue associated with the component.

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ I believe you should rather be using the FactoMineR package (which is what the authors use) to understand the above statement: FactoMineR::PCA doesn't return the loadings but the coordinates of the variables on each PC. Usually, we say that the eigenvector times the square root of the eigenvalue gives the component loadings which can be interpreted as the correlation of each item with the principal component. More information in this great post by @amoeba. $\endgroup$ Commented Oct 22, 2020 at 19:31
  • $\begingroup$ On loadings and eigenvectors of PCA stats.stackexchange.com/q/143905/3277 $\endgroup$ Commented Oct 24, 2020 at 15:53
  • $\begingroup$ @chl, I added the FactoMineR package code as requested, but the statement gets more confusing by the word divided by, what could be the better wording for that then? $\endgroup$ Commented Oct 24, 2020 at 16:03
  • $\begingroup$ @ttnphns, I am aware of that excellent post on this confusion between the two, and how prcomp() output $rotation refers to loadings when they are actually eigenvectors (direction only devoid of magnitude reflected by the variance). But would you agree with the statement made by the author? If not, how would you re-phrase it correctly? – $\endgroup$ Commented Oct 24, 2020 at 16:17
  • 1
    $\begingroup$ I believe the confusion comes from the fact that the authors of the FactoMineR package use terminology related to PCA "à la française" (French school of data analysis), according to which factorial coordinates reflect correlations between variables and factor axis (or components). $\endgroup$ Commented Oct 24, 2020 at 18:24

1 Answer 1

Reset to default
3
$\begingroup$

I'm answering only to the citation at the beginning of the question. I did not consider the analysis in R provided in the question.

I admit that the author of the first passage may have made some confusion in terminology. Let us define properly.

$v$ is the eigenvector's values. It is the eigenvector from eigencecomposition of the covariance matrix of the variables or, equivalently, the right eigenvector from SVD decomposition of the data matrix. Eigenvector values are the standardized coordinates of the variables as vectors in the space of the principal components, i.e. on a biplot or loading plot - see last paragraph here.)

$\lambda$ is the corresponding eigenvalue (the component's variance).

$a=v \sqrt{\lambda}$ is the loading vector. Loadings $a$ are the coefficients of the linear combination predicting a variable by unit-scaled principal components. $V_i= a_{i1}F_1+a_{i2}F_2...$.

While the coefficients of the linear combination to compute unit-scaled principal component values (scores), $F_j= b_{1j}V_1+b_{2j}V_2...$, are given by

$b=a/\lambda=\frac{v\sqrt{\lambda}}{\lambda}=v/\sqrt{\lambda}$

(see here, "Computation of B in PCA: ... When component loadings are extracted but not rotated (...) amounts to simply dividing each column of A by the respective eigenvalue - the component's variance").

So you have eigenvector divided by the square root of the corresponding eigenvalue. The original citation was:

Loadings are interpreted as the coefficients of the linear combination of the initial variables from which the principal components are constructed. ... the loadings are equal to the coordinates of the variables divided by the square root of the eigenvalue associated with the component.

Then, if my interpretation is correct, that passage should sound like:

Component score coefficients are the coefficients of the linear combination of the initial variables from which the principal components [component scores] are constructed. ... the coefficients are equal to the coordinates of the variables [the eigenvectors] divided by the square root of the eigenvalue associated with the component.

In this case, the author seems to mean component score coefficients by the term "loadings".

This is so if by "coordinates of the variables" the author meant eigenvectors (standardized = normalized = unit-scaled coordinates of the variables) and implied the components themselves to be unit-scaled (as we often think in factor analysis context).

But if by "coordinates of the variables" the author meant loadings (variance-scaled or full-scaled coordinates of the variables) and implied the components themselves to be variance-scaled or full-scaled (as we often think in PCA context), then we are in the situation of tautology:

Since $a=v \sqrt{\lambda}$, then $v=a/ \sqrt{\lambda}$,

and also we know that then eigenvector values are the coefficients $b$ to compute the full-scaled principal components from the input variables. If that is what the author meant, then the passage should sound like:

Eigenvectors are the component score coefficients, the coefficients of the linear combination of the initial variables from which the principal components [component scores] are constructed. ... the coefficients are equal to the coordinates of the variables [the loadings] divided by the square root of the eigenvalue associated with the component.

In this case, the author seems to mean eigenvector entries by the term "loadings".

I might recommend to read the paragraph "Digression" in my answer about the similarities and differences between PCA and Factor analysis.

Improve this answer
$\endgroup$
3
  • $\begingroup$ from the above R analysis, by numbers I would confirm that per FactoMineR package, the output$var$coord means loadings (variance-scaled or full-scaled coordinates of the variables) then according to your second premise we are in the situation of tautalogy... $\endgroup$ Commented Oct 27, 2020 at 16:22
  • $\begingroup$ can I say that the $B$ matrix would be the change of basis matrix that would transform my newly obtained data in the experiment to the newly projected PCA space spanned by the first plane for example of PC1 and PC2? So $B$ is $V$ when $XV$= $U \Sigma$ = PC scores. $\endgroup$ Commented Oct 27, 2020 at 16:24
  • $\begingroup$ Yes, correct. PC scores (unstandardized, i.e. with their native variance) are computed directly by eigenvectors, so B=V then. $\endgroup$ Commented Oct 27, 2020 at 18:39

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.