Principal Component Analysis (PCA) - Statistics Graduate Level

1. Which statement correctly describes the first principal-component directions under the two analyses?
   Answer: Covariance PCA is dominated by the first variable because its variance is far larger, whereas standardized PCA gives the direction $2^{-1/2}(1,1)^T$ up to sign because the correlation is $0.6$. Whenever you see a PCA question comparing covariance-based and correlation-based (standardized) analyses, your first instinct should be to ask: do the variables have very different variances? If so, the two approaches can give dramatically different results. Here, $\Sigma = \begin{pmatrix}100 & 6 \\ 6 & 1\end{pmatrix}$. The first variable has variance 100, the second has variance 1 — a 100:1 ratio. In covariance PCA, the eigenvector corresponding to the largest eigenvalue is pulled almost entirely toward the direction of the high-variance variable. Intuitively, the first PC must "explain" as much total variance as possible, so it aligns nearly with the $(1, 0)^T$ axis. The first variable dominates completely. For standardized PCA, you instead analyze the correlation matrix. Dividing each variable by its standard deviation (10 and 1, respectively) gives $R = \begin{pmatrix}1 & 0.6 \\ 0.6 & 1\end{pmatrix}$, since $\rho = 6/(10 \cdot 1) = 0.6$. For a symmetric $2\times 2$ correlation matrix with positive off-diagonal entry, the leading eigenvector is $2^{-1/2}(1,1)^T$ — the direction of maximum variance runs along the positive diagonal. This confirms answer C is correct. A is wrong because it ignores how unequal variances distort covariance PCA — the direction $2^{-1/2}(1,1)^T$ is not the covariance PC1. B incorrectly claims the covariance is dominated by the second variable (variance = 1) and misidentifies the correlation as negative — it is $+0.6$, not negative. D is wrong because standardizing changes the matrix being analyzed; eigenvectors of $\Sigma$ are generally not preserved after standardization. Study tip: Always compute the correlation $\rho = \sigma_{12}/(\sigma_1 \sigma_2)$ explicitly — it often reveals sign and structure that raw covariance obscures, and it's the key to predicting standardized-PCA behavior.
2. What are the correlations of $X_1$ and $X_2$ with the first principal-component score?
   Answer: $0.90$ and $-0.60$, because each loading entry is multiplied by $\sqrt{\lambda_1}$. When working with PCA on a correlation matrix, you need to distinguish between two related but different quantities: the loading vector entries (eigenvector components) and the correlations between original variables and principal component scores. These are not the same thing, and confusing them is the most common trap on this topic. The correct relationship is: $\text{Cor}(X_j, PC_1) = v_{j1} \cdot \sqrt{\lambda_1}$ This formula arises because the principal component score has variance equal to $\lambda_1$, so when you standardize to compute a correlation, a factor of $\sqrt{\lambda_1}$ enters. For this problem: $\text{Cor}(X_1, PC_1) = 0.60 \times \sqrt{2.25} = 0.60 \times 1.50 = 0.90$, and $\text{Cor}(X_2, PC_1) = -0.40 \times 1.50 = -0.60$. This confirms B is correct. A is the classic misconception — treating eigenvector entries directly as correlations. The eigenvector entries are unit-normalized loadings, not correlations. They become correlations only after scaling by $\sqrt{\lambda_1}$. C inverts the scaling, dividing by $\sqrt{\lambda_1}$ instead of multiplying. This has no statistical interpretation in this context and would actually shrink the relationship toward zero, which makes no sense directionally. D multiplies by $\lambda_1$ itself rather than its square root, conflating variance ($\lambda_1$) with standard deviation ($\sqrt{\lambda_1}$). Correlations are bounded by $[-1, 1]$, and this answer produces $1.35$, which should immediately signal an error. A reliable memory aid: **correlations use $\sqrt{\lambda}$, variances use $\lambda$**. Always ask yourself which quantity the question is requesting before applying any formula.
3. Which conclusion about the population principal components is correct?
   Answer: Any unit vector in $\operatorname{span}(q_1,q_2)$ can serve as a first loading, while that two-dimensional principal subspace is unique. When eigenvalues repeat in a spectral decomposition, the geometry of principal components changes fundamentally — and that's exactly what this question tests. The key is distinguishing between the principal subspace and the individual loading vectors. Here, $\Sigma = Q\,\text{diag}(5,5,1)Q^T$ has a repeated eigenvalue of 5 for the first two components. This means any unit vector lying in $\text{span}(q_1, q_2)$ satisfies $\Sigma v = 5v$, making it a valid eigenvector with variance 5. There is no mathematical basis for preferring $q_1$ over $q_2$, or any rotation within that plane. However, the two-dimensional subspace itself is uniquely determined — it's the eigenspace corresponding to eigenvalue 5, which is geometrically fixed regardless of which basis you choose for it. This is exactly what B describes, making it correct. A is wrong because it assumes $q_1$ is the unique first loading. With a repeated eigenvalue, the entire eigenspace qualifies — not just the columns of $Q$ as written. The sign-reversal comment is a red herring that distracts from the real issue of non-uniqueness. C is wrong on two counts: first, any unit vector in $\text{span}(q_1,q_2)$ (not just $q_1$ or $q_2$) is valid; second, rotations within the eigenspace preserve uncorrelatedness of component scores, so the correlation argument is false. D reverses the truth. The subspace is identifiable and unique; it's the individual loading vectors within that subspace that are not. Study tip: On any PCA question, immediately check whether eigenvalues repeat. Repeated eigenvalues → the subspace is unique but individual loadings within it are not. Distinct eigenvalues → loadings are unique up to sign.
4. Which expression is the sample covariance matrix of the columns of $Z$?
   Answer: $\operatorname{diag}(\lambda_1,\ldots,\lambda_k)$, because PCA scores are uncorrelated but not generally standardized. Whenever PCA appears on a graduate exam, anchor yourself to one key calculation: the sample covariance of the projected scores. If $Z = XV_k$, then the sample covariance of $Z$ is $\frac{Z^TZ}{n-1} = \frac{V_k^T X^T X V_k}{n-1} = V_k^T S V_k$. Now apply the spectral property: since the columns of $V_k$ are orthonormal eigenvectors of $S$, we have $SV_k = V_k \operatorname{diag}(\lambda_1,\ldots,\lambda_k)$, which gives $V_k^T S V_k = \operatorname{diag}(\lambda_1,\ldots,\lambda_k)$. This is answer A — the scores are uncorrelated (off-diagonal entries vanish because eigenvectors are orthogonal), and each component has variance equal to its eigenvalue, not 1. Answer B is tempting but wrong. Orthonormality of the loading vectors $V_k$ means $V_k^TV_k = I$, not that the resulting scores have unit variance. The variance of each score component is $\lambda_i$, which equals 1 only if the data happen to be pre-standardized to that scale. Answer C describes a matrix in the original $p$-dimensional space — it's actually the reconstruction $V_k \operatorname{diag}(\lambda_1,\ldots,\lambda_k)V_k^T$, which is the rank-$k$ approximation of $S$, not the covariance of the scores themselves. Answer D inverts the eigenvalues, which would arise in whitening (scaling by $\lambda_i^{-1/2}$), not in standard PCA projection. Your study tip: memorize the chain $\text{Cov}(Z) = V_k^T S V_k = \operatorname{diag}(\lambda_1,\ldots,\lambda_k)$. Questions about PCA scores almost always hinge on whether you recognize that orthonormality kills the cross-terms but does not standardize the variances.
5. Which statement is guaranteed to be true?
   Answer: The covariance matrix has at least four zero eigenvalues because the centered data matrix has rank at most four. Whenever you see a PCA question involving a data matrix with more variables than observations, your first instinct should be to think about matrix rank, not just the number of variables. Here's the key insight: start with your centered data matrix $\mathbf{X}$, which is $n \times p = 5 \times 8$. Centering each column (subtracting the sample mean) imposes one linear constraint — the columns now sum to zero — which reduces the row rank by at most 1. Therefore, $\text{rank}(\mathbf{X}) \leq n - 1 = 4$. The sample covariance matrix is proportional to $\mathbf{X}^\top \mathbf{X}$, and a fundamental theorem of linear algebra tells us $\text{rank}(\mathbf{X}^\top \mathbf{X}) = \text{rank}(\mathbf{X}) \leq 4$. Since $\mathbf{X}^\top \mathbf{X}$ is an $8 \times 8$ matrix with rank at most 4, it must have at least $8 - 4 = 4$ zero eigenvalues. This confirms C is guaranteed to be true. A is wrong because it claims exactly three zero eigenvalues using only the difference $p - n = 3$. This ignores the centering constraint and conflates $n$ with $n-1$. B is wrong in the opposite direction — it claims centering removes only one direction, leaving at most four zero eigenvalues. But centering reduces available rank to $n-1=4$, guaranteeing at least four zeros, not at most. D is wrong because while $n$ limits rank, the rank could be less than 4 if columns of $\mathbf{X}$ are linearly dependent, so you cannot guarantee exactly four positive eigenvalues. Study tip: Memorize this chain — centering an $n \times p$ matrix caps rank at $\min(n-1, p)$, which directly bounds how many nonzero eigenvalues the covariance matrix can have. "At least" vs. "exactly" language is almost always a trap.
6. Which pair gives the expected squared reconstruction error and the proportion of total variance retained by this approximation?
   Answer: The reconstruction error is $5$, and the retained proportion is $9/14$. When working with PCA, two quantities matter: how much variance the retained components explain, and how much reconstruction error the discarded components introduce. The key insight is that eigenvalues represent variance along each principal component, so the total variance is their sum, and the reconstruction error from a rank-one approximation equals the sum of the discarded eigenvalues. Here, the total variance is $\lambda_1 + \lambda_2 + \lambda_3 = 9 + 4 + 1 = 14$. Projecting onto only the first principal component discards the second and third components, so the expected squared reconstruction error is $\lambda_2 + \lambda_3 = 4 + 1 = 5$. The proportion of variance retained is the variance captured by the first component divided by the total: $\lambda_1 / (\lambda_1 + \lambda_2 + \lambda_3) = 9/14$. This matches answer A, which is correct. Answer B swaps the two quantities — it reports $9$ as the reconstruction error (which is actually $\lambda_1$, the retained variance, not the lost variance) and $5/14$ as the retained proportion (which is actually the lost fraction). This is a classic reversal trap. Answer C gets the reconstruction error right ($5$) but uses $13$ as the denominator instead of $14$, suggesting the student mistakenly excluded one eigenvalue from the total — perhaps thinking the retained component shouldn't count toward the denominator. Answer D reports a reconstruction error of $4$, which is only $\lambda_2$, incorrectly omitting $\lambda_3$ from the discarded components. A reliable rule of thumb: reconstruction error = sum of dropped eigenvalues; retained proportion = sum of kept eigenvalues ÷ total. Never confuse what you keep with what you lose.
7. What is the first principal component of the augmented data, up to an overall sign?
   Answer: It is $2^{-1/2}(X_1+X_3)$ and explains two-thirds of the augmented total variance. When you encounter PCA on redundant or duplicated variables, start by writing out the covariance matrix explicitly — that's where everything becomes clear. Since $X_1, X_2, X_3$ are standardized and $X_3 = X_1$, the covariance matrix is: $\Sigma = \begin{pmatrix} 1 & 0 & 1 \\ 0 & 1 & 0 \\ 1 & 0 & 1 \end{pmatrix}$ The total variance is $\text{tr}(\Sigma) = 3$. To find principal components, solve for eigenvalues. Notice $X_2$ is uncorrelated with both others, so it contributes one eigenvalue of 1. For the $X_1$-$X_3$ block $\begin{pmatrix}1&1\\1&1\end{pmatrix}$, the eigenvalues are 2 and 0. So the full eigenvalues are 2, 1, 0, in decreasing order. The first PC corresponds to eigenvalue 2, with eigenvector $(1/\sqrt{2}, 0, 1/\sqrt{2})^T$, giving **$2^{-1/2}(X_1 + X_3)$**. It explains $2/3$ of total variance. This confirms A is correct. B is wrong because $X_2$ corresponds to eigenvalue 1, the second PC, explaining only $1/3$ (not $1/2$) of variance. C describes the direction $2^{-1/2}(X_1 - X_3)$, which corresponds to eigenvalue 0 — a completely degenerate direction with no variance at all, since $X_1 - X_3 = 0$ identically. D is wrong because the equal-weight combination is not an eigenvector of $\Sigma$, and no single component can explain all variance when the eigenvalue 0 doesn't eliminate it entirely. When variables are duplicated in PCA, the duplicate pair always produces one direction of doubled variance and one of zero variance — the first PC "absorbs" the redundancy rather than ignoring it.
8. Which pair gives the correct first-component score and the corresponding rank-one reconstruction in the original coordinates?
   Answer: The score is $4.8$, and the reconstruction is $(5.84,3.88)^T$. Whenever you see a PCA reconstruction question, remember there are two distinct steps: computing the score (projection), then reconstructing in original space (unprojection). Confusing these steps is exactly what the distractors exploit. Step 1 – Center the observation. PCA operates on mean-centered data, so you first subtract the training mean: $x - \mu = (5-2,\ 5-1)^T = (3,\ 4)^T$. Step 2 – Compute the score. Project the centered vector onto the first loading: $t_1 = v_1^T(x-\mu) = (0.8)(3) + (0.6)(4) = 2.4 + 2.4 = 4.8$. Step 3 – Reconstruct. The rank-one approximation in original coordinates adds the mean back: $\hat{x} = \mu + t_1 v_1 = (2,1)^T + 4.8(0.8,0.6)^T = (2+3.84,\ 1+2.88)^T = (5.84,\ 3.88)^T$. This confirms D is correct. Now for the distractors. A would be right only if the centered observation were orthogonal to $v_1$, giving a zero score — that's not the case here. B scores $7.0$, which comes from projecting the raw $x$ without centering: $(0.8)(5)+(0.6)(5)=7$, a classic mean-subtraction omission error. C gets the score right ($4.8$) but forgets to add the training mean back during reconstruction, reporting only $t_1 v_1 = (3.84,2.88)^T$ as if the origin were at zero rather than $\mu$. Study tip: Memorize the full PCA cycle as center → project → uncenter. On exam questions, explicitly write out the centering and re-adding of the mean — skipping either step maps directly to a wrong answer choice.
9. At the population level, what is the principal consequence of this component-selection rule?
   Answer: The retained component has regression coefficient zero, so the procedure discards the predictor direction containing the signal. Whenever you encounter a principal component regression (PCR) question, your first instinct should be to ask: does variance in the predictors correspond to variance relevant to the response? PCR selects components by unsupervised criterion — it knows nothing about $Y$. Here, the predictors are uncorrelated with variances $100$ and $1$, so the first principal component aligns almost entirely with $X_1$ (the high-variance direction), and the second aligns almost entirely with $X_2$. But the true model is $Y = X_2 + \varepsilon$, meaning all the signal lives in the low-variance direction. When you regress $Y$ onto only the first component — which is essentially $X_1$ — the population regression coefficient is zero, because $X_1$ is uncorrelated with $Y$ by construction. PCR retains a predictor direction with zero predictive power and discards the one direction that matters. That confirms D as correct. A is the classic PCR misconception: explaining predictor variance does not imply explaining response variance. These are completely separate quantities, and this question is a textbook counterexample. B describes how a supervised method like partial least squares (PLS) operates — PLS does rotate loadings toward the response, but PCR never involves $Y$ in component construction. C is false because orthogonal components can differ dramatically in predictive relevance, as this example demonstrates; OLS uses all components and would recover the signal, but PCR with truncation does not. Your study tip: whenever you see PCR, mentally flag that component selection is unsupervised — high predictor variance and high response relevance are independent properties that can be dangerously misaligned.

Start this quiz