Statistics Graduate Level Quiz: Principal Component Analysis Pca
9 questions · exam conditions
0:00
Principal Component Analysis PcaQuestion 1 of 9

The covariance matrix of a centered random vector has ordered eigenvalues λ1=9\lambda_1=9, λ2=4\lambda_2=4, and λ3=1\lambda_3=1. A rank-one PCA approximation is formed by projecting onto the first principal component.

Which pair gives the expected squared reconstruction error and the proportion of total variance retained by this approximation?

The reconstruction error is 55, and the retained proportion is 9/149/14.
The reconstruction error is 99, and the retained proportion is 5/145/14.
The reconstruction error is 55, and the retained proportion is 9/139/13.
The reconstruction error is 44, and the retained proportion is 10/1410/14.
← Back to quizzes

Statistics Graduate Level Quiz

Statistics Graduate Level Quiz: Principal Component Analysis Pca

Practice Principal Component Analysis Pca in Statistics Graduate Level with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on Principal Component Analysis Pca, giving you a quick way to practice the rules, question types, and explanations that matter most for Statistics Graduate Level.

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

The covariance matrix of a centered random vector has ordered eigenvalues λ1=9\lambda_1=9, λ2=4\lambda_2=4, and λ3=1\lambda_3=1. A rank-one PCA approximation is formed by projecting onto the first principal component.

Which pair gives the expected squared reconstruction error and the proportion of total variance retained by this approximation?

  1. The reconstruction error is 55, and the retained proportion is 9/149/14. (correct answer)
  2. The reconstruction error is 99, and the retained proportion is 5/145/14.
  3. The reconstruction error is 55, and the retained proportion is 9/139/13.
  4. The reconstruction error is 44, and the retained proportion is 10/1410/14.
Explanation: When working with PCA, two quantities matter: how much variance the retained components explain, and how much reconstruction error the discarded components introduce. The key insight is that eigenvalues represent variance along each principal component, so the total variance is their sum, and the reconstruction error from a rank-one approximation equals the sum of the discarded eigenvalues. Here, the total variance is λ1+λ2+λ3=9+4+1=14\lambda_1 + \lambda_2 + \lambda_3 = 9 + 4 + 1 = 14. Projecting onto only the first principal component discards the second and third components, so the expected squared reconstruction error is λ2+λ3=4+1=5\lambda_2 + \lambda_3 = 4 + 1 = 5. The proportion of variance retained is the variance captured by the first component divided by the total: λ1/(λ1+λ2+λ3)=9/14\lambda_1 / (\lambda_1 + \lambda_2 + \lambda_3) = 9/14. This matches answer A, which is correct. Answer B swaps the two quantities — it reports 99 as the reconstruction error (which is actually λ1\lambda_1, the retained variance, not the lost variance) and 5/145/14 as the retained proportion (which is actually the lost fraction). This is a classic reversal trap. Answer C gets the reconstruction error right (55) but uses 1313 as the denominator instead of 1414, suggesting the student mistakenly excluded one eigenvalue from the total — perhaps thinking the retained component shouldn't count toward the denominator. Answer D reports a reconstruction error of 44, which is only λ2\lambda_2, incorrectly omitting λ3\lambda_3 from the discarded components. A reliable rule of thumb: reconstruction error = sum of dropped eigenvalues; retained proportion = sum of kept eigenvalues ÷ total. Never confuse what you keep with what you lose.

Question 2

Let XX be an n×pn\times p centered data matrix with sample covariance S=XTX/(n−1)S=X^TX/(n-1). Let VkV_k contain orthonormal eigenvectors of SS associated with eigenvalues λ1,…,λk\lambda_1,\ldots,\lambda_k, and define the score matrix Z=XVkZ=XV_k.

Which expression is the sample covariance matrix of the columns of ZZ?

  1. diag⁡(λ1,…,λk)\operatorname{diag}(\lambda_1,\ldots,\lambda_k), because PCA scores are uncorrelated but not generally standardized. (correct answer)
  2. IkI_k, because orthonormal loading vectors necessarily produce scores with unit sample variance.
  3. Vkdiag⁡(λ1,…,λk)VkTV_k\operatorname{diag}(\lambda_1,\ldots,\lambda_k)V_k^T, because the score covariance remains in the original coordinates.
  4. diag⁡(λ1−1,…,λk−1)\operatorname{diag}(\lambda_1^{-1},\ldots,\lambda_k^{-1}), because projection onto eigenvectors inverts the component variances.
Explanation: Whenever PCA appears on a graduate exam, anchor yourself to one key calculation: the sample covariance of the projected scores. If Z=XVkZ = XV_k, then the sample covariance of ZZ is ZTZn−1=VkTXTXVkn−1=VkTSVk\frac{Z^TZ}{n-1} = \frac{V_k^T X^T X V_k}{n-1} = V_k^T S V_k. Now apply the spectral property: since the columns of VkV_k are orthonormal eigenvectors of SS, we have SVk=Vkdiag⁡(λ1,…,λk)SV_k = V_k \operatorname{diag}(\lambda_1,\ldots,\lambda_k), which gives VkTSVk=diag⁡(λ1,…,λk)V_k^T S V_k = \operatorname{diag}(\lambda_1,\ldots,\lambda_k). This is answer A — the scores are uncorrelated (off-diagonal entries vanish because eigenvectors are orthogonal), and each component has variance equal to its eigenvalue, not 1. Answer B is tempting but wrong. Orthonormality of the loading vectors VkV_k means VkTVk=IV_k^TV_k = I, not that the resulting scores have unit variance. The variance of each score component is λi\lambda_i, which equals 1 only if the data happen to be pre-standardized to that scale. Answer C describes a matrix in the original pp-dimensional space — it's actually the reconstruction Vkdiag⁡(λ1,…,λk)VkTV_k \operatorname{diag}(\lambda_1,\ldots,\lambda_k)V_k^T, which is the rank-kk approximation of SS, not the covariance of the scores themselves. Answer D inverts the eigenvalues, which would arise in whitening (scaling by λi−1/2\lambda_i^{-1/2}), not in standard PCA projection. Your study tip: memorize the chain Cov(Z)=VkTSVk=diag⁡(λ1,…,λk)\text{Cov}(Z) = V_k^T S V_k = \operatorname{diag}(\lambda_1,\ldots,\lambda_k). Questions about PCA scores almost always hinge on whether you recognize that orthonormality kills the cross-terms but does not standardize the variances.

Question 3

PCA is performed on the correlation matrix of standardized variables. The first eigenvalue is λ1=2.25\lambda_1=2.25, and the first loading vector has entries v11=0.60v_{11}=0.60 and v21=−0.40v_{21}=-0.40 for variables X1X_1 and X2X_2, respectively.

What are the correlations of X1X_1 and X2X_2 with the first principal-component score?

  1. 0.600.60 and −0.40-0.40, because correlations equal loading-vector entries for standardized variables.
  2. 0.900.90 and −0.60-0.60, because each loading entry is multiplied by λ1\sqrt{\lambda_1}. (correct answer)
  3. 0.400.40 and −0.27-0.27, because each loading entry is divided by λ1\sqrt{\lambda_1}.
  4. 1.351.35 and −0.90-0.90, because each loading entry is multiplied directly by λ1\lambda_1.
Explanation: When working with PCA on a correlation matrix, you need to distinguish between two related but different quantities: the loading vector entries (eigenvector components) and the correlations between original variables and principal component scores. These are not the same thing, and confusing them is the most common trap on this topic. The correct relationship is: Cor(Xj,PC1)=vj1⋅λ1\text{Cor}(X_j, PC_1) = v_{j1} \cdot \sqrt{\lambda_1} This formula arises because the principal component score has variance equal to λ1\lambda_1, so when you standardize to compute a correlation, a factor of λ1\sqrt{\lambda_1} enters. For this problem: Cor(X1,PC1)=0.60×2.25=0.60×1.50=0.90\text{Cor}(X_1, PC_1) = 0.60 \times \sqrt{2.25} = 0.60 \times 1.50 = 0.90, and Cor(X2,PC1)=−0.40×1.50=−0.60\text{Cor}(X_2, PC_1) = -0.40 \times 1.50 = -0.60. This confirms B is correct. A is the classic misconception — treating eigenvector entries directly as correlations. The eigenvector entries are unit-normalized loadings, not correlations. They become correlations only after scaling by λ1\sqrt{\lambda_1}. C inverts the scaling, dividing by λ1\sqrt{\lambda_1} instead of multiplying. This has no statistical interpretation in this context and would actually shrink the relationship toward zero, which makes no sense directionally. D multiplies by λ1\lambda_1 itself rather than its square root, conflating variance (λ1\lambda_1) with standard deviation (λ1\sqrt{\lambda_1}). Correlations are bounded by [−1,1][-1, 1], and this answer produces 1.351.35, which should immediately signal an error. A reliable memory aid: correlations use λ\sqrt{\lambda}, variances use λ\lambda. Always ask yourself which quantity the question is requesting before applying any formula.

Question 4

Two centered variables have covariance matrix $$\Sigma=\begin{pmatrix}100&6\6&1\end{pmatrix}

Which statement correctly describes the first principal-component directions under the two analyses?

  1. Both analyses give the direction 2−1/2(1,1)T2^{-1/2}(1,1)^T because a positive covariance between the variables produces the same leading eigenvector regardless of their individual variances.
  2. Covariance PCA is dominated by the second variable because the covariance is small relative to its variance, whereas standardized PCA gives the direction 2−1/2(1,−1)T2^{-1/2}(1,-1)^T up to sign because the correlation is negative.
  3. Covariance PCA is dominated by the first variable because its variance is far larger, whereas standardized PCA gives the direction 2−1/2(1,1)T2^{-1/2}(1,1)^T up to sign because the correlation is 0.60.6. (correct answer)
  4. Both analyses are dominated by the first variable because dividing each variable by its standard deviation preserves all eigenvectors of the original covariance matrix.
Explanation: Whenever you see a PCA question comparing covariance-based and correlation-based (standardized) analyses, your first instinct should be to ask: do the variables have very different variances? If so, the two approaches can give dramatically different results. Here, Σ=(100661)\Sigma = \begin{pmatrix}100 & 6 \\ 6 & 1\end{pmatrix} . The first variable has variance 100, the second has variance 1 — a 100:1 ratio. In covariance PCA, the eigenvector corresponding to the largest eigenvalue is pulled almost entirely toward the direction of the high-variance variable. Intuitively, the first PC must "explain" as much total variance as possible, so it aligns nearly with the (1,0)T(1, 0)^T axis. The first variable dominates completely. For standardized PCA, you instead analyze the correlation matrix. Dividing each variable by its standard deviation (10 and 1, respectively) gives R=(10.60.61)R = \begin{pmatrix}1 & 0.6 \\ 0.6 & 1\end{pmatrix} , since ρ=6/(10⋅1)=0.6\rho = 6/(10 \cdot 1) = 0.6. For a symmetric 2×22\times 2 correlation matrix with positive off-diagonal entry, the leading eigenvector is 2−1/2(1,1)T2^{-1/2}(1,1)^T — the direction of maximum variance runs along the positive diagonal. This confirms answer C is correct. A is wrong because it ignores how unequal variances distort covariance PCA — the direction 2−1/2(1,1)T2^{-1/2}(1,1)^T is not the covariance PC1. B incorrectly claims the covariance is dominated by the second variable (variance = 1) and misidentifies the correlation as negative — it is +0.6+0.6, not negative. D is wrong because standardizing changes the matrix being analyzed; eigenvectors of Σ\Sigma are generally not preserved after standardization. Study tip: Always compute the correlation ρ=σ12/(σ1σ2)\rho = \sigma_{12}/(\sigma_1 \sigma_2) explicitly — it often reveals sign and structure that raw covariance obscures, and it's the key to predicting standardized-PCA behavior.

Question 5

Predictors X1X_1 and X2X_2 are centered, mutually uncorrelated, and have variances 100100 and 11, respectively. The response satisfies Y=X2+εY=X_2+\varepsilon, where ε\varepsilon is independent of both predictors and has mean zero. Principal component regression retains only the first predictor principal component because it explains 100/101100/101 of the predictor variance.

At the population level, what is the principal consequence of this component-selection rule?

  1. The retained component predicts well because explaining nearly all predictor variance guarantees explaining nearly all response variation.
  2. The response rotates the first loading toward X2X_2, so supervised fitting recovers the low-variance signal automatically.
  3. The retained component reproduces ordinary least squares because orthogonal predictor components cannot differ in predictive relevance.
  4. The retained component has regression coefficient zero, so the procedure discards the predictor direction containing the signal. (correct answer)
Explanation: Whenever you encounter a principal component regression (PCR) question, your first instinct should be to ask: does variance in the predictors correspond to variance relevant to the response? PCR selects components by unsupervised criterion — it knows nothing about YY. Here, the predictors are uncorrelated with variances 100100 and 11, so the first principal component aligns almost entirely with X1X_1 (the high-variance direction), and the second aligns almost entirely with X2X_2. But the true model is Y=X2+εY = X_2 + \varepsilon, meaning all the signal lives in the low-variance direction. When you regress YY onto only the first component — which is essentially X1X_1 — the population regression coefficient is zero, because X1X_1 is uncorrelated with YY by construction. PCR retains a predictor direction with zero predictive power and discards the one direction that matters. That confirms D as correct. A is the classic PCR misconception: explaining predictor variance does not imply explaining response variance. These are completely separate quantities, and this question is a textbook counterexample. B describes how a supervised method like partial least squares (PLS) operates — PLS does rotate loadings toward the response, but PCR never involves YY in component construction. C is false because orthogonal components can differ dramatically in predictive relevance, as this example demonstrates; OLS uses all components and would recover the signal, but PCR with truncation does not. Your study tip: whenever you see PCR, mentally flag that component selection is unsupervised — high predictor variance and high response relevance are independent properties that can be dangerously misaligned.

Question 6

Two standardized variables X1X_1 and X2X_2 are uncorrelated. An analyst creates a third standardized variable X3=X1X_3=X_1 and performs covariance-based PCA on (X1,X2,X3)T(X_1,X_2,X_3)^T.

What is the first principal component of the augmented data, up to an overall sign?

  1. It is 2−1/2(X1+X3)2^{-1/2}(X_1+X_3) and explains two-thirds of the augmented total variance. (correct answer)
  2. It is X2X_2 and explains one-half of the augmented total variance.
  3. It is 2−1/2(X1−X3)2^{-1/2}(X_1-X_3) and explains two-thirds of the augmented total variance.
  4. It is 3−1/2(X1+X2+X3)3^{-1/2}(X_1+X_2+X_3) and explains all of the augmented total variance.
Explanation: When you encounter PCA on redundant or duplicated variables, start by writing out the covariance matrix explicitly — that's where everything becomes clear. Since X1,X2,X3X_1, X_2, X_3 are standardized and X3=X1X_3 = X_1, the covariance matrix is: The total variance is tr(Σ)=3\text{tr}(\Sigma) = 3. To find principal components, solve for eigenvalues. Notice X2X_2 is uncorrelated with both others, so it contributes one eigenvalue of 1. For the X1X_1-X3X_3 block $$ \begin{pmatrix}1&1\1&1\end{pmatrix} The first PC corresponds to eigenvalue 2, with eigenvector $$(1/\sqrt{2}, 0, 1/\sqrt{2})^T$$, giving **$$2^{-1/2}(X_1 + X_3)$$**. It explains $$2/3$$ of total variance. This confirms **A** is correct. **B** is wrong because $$X_2$$ corresponds to eigenvalue 1, the *second* PC, explaining only $$1/3$$ (not $$1/2$$) of variance. **C** describes the direction $$2^{-1/2}(X_1 - X_3)$$, which corresponds to eigenvalue **0** — a completely degenerate direction with no variance at all, since $$X_1 - X_3 = 0$$ identically. **D** is wrong because the equal-weight combination is not an eigenvector of $$\Sigma$$, and no single component can explain all variance when the eigenvalue 0 doesn't eliminate it entirely. When variables are duplicated in PCA, the duplicate pair always produces one direction of doubled variance and one of zero variance — the first PC "absorbs" the redundancy rather than ignoring it.

Question 7

A covariance-based PCA model for two variables was fitted using training mean μ=(2,1)T\mu=(2,1)^T. Its first unit loading is v1=(0.8,0.6)Tv_1=(0.8,0.6)^T. A new observation is x=(5,5)Tx=(5,5)^T, and it is to be represented using only the first component.

Which pair gives the correct first-component score and the corresponding rank-one reconstruction in the original coordinates?

  1. The score is 00, and the reconstruction is the training mean (2,1)T(2,1)^T.
  2. The score is 7.07.0, and the reconstruction is (5.60,4.20)T(5.60,4.20)^T.
  3. The score is 4.84.8, and the reconstruction is (3.84,2.88)T(3.84,2.88)^T.
  4. The score is 4.84.8, and the reconstruction is (5.84,3.88)T(5.84,3.88)^T. (correct answer)
Explanation: Whenever you see a PCA reconstruction question, remember there are two distinct steps: computing the score (projection), then reconstructing in original space (unprojection). Confusing these steps is exactly what the distractors exploit. Step 1 – Center the observation. PCA operates on mean-centered data, so you first subtract the training mean: x−μ=(5−2, 5−1)T=(3, 4)Tx - \mu = (5-2,\ 5-1)^T = (3,\ 4)^T. Step 2 – Compute the score. Project the centered vector onto the first loading: t1=v1T(x−μ)=(0.8)(3)+(0.6)(4)=2.4+2.4=4.8t_1 = v_1^T(x-\mu) = (0.8)(3) + (0.6)(4) = 2.4 + 2.4 = 4.8. Step 3 – Reconstruct. The rank-one approximation in original coordinates adds the mean back: x^=μ+t1v1=(2,1)T+4.8(0.8,0.6)T=(2+3.84, 1+2.88)T=(5.84, 3.88)T\hat{x} = \mu + t_1 v_1 = (2,1)^T + 4.8(0.8,0.6)^T = (2+3.84,\ 1+2.88)^T = (5.84,\ 3.88)^T. This confirms D is correct. Now for the distractors. A would be right only if the centered observation were orthogonal to v1v_1, giving a zero score — that's not the case here. B scores 7.07.0, which comes from projecting the raw xx without centering: (0.8)(5)+(0.6)(5)=7(0.8)(5)+(0.6)(5)=7, a classic mean-subtraction omission error. C gets the score right (4.84.8) but forgets to add the training mean back during reconstruction, reporting only t1v1=(3.84,2.88)Tt_1 v_1 = (3.84,2.88)^T as if the origin were at zero rather than μ\mu. Study tip: Memorize the full PCA cycle as center → project → uncenter. On exam questions, explicitly write out the centering and re-adding of the mean — skipping either step maps directly to a wrong answer choice.

Question 8

A data matrix contains observations on p=8p=8 variables for n=5n=5 subjects. Each variable is centered using its sample mean, and PCA is performed on the resulting sample covariance matrix. No additional assumptions about linear dependence are made.

Which statement is guaranteed to be true?

  1. The covariance matrix has exactly three zero eigenvalues because there are three more variables than subjects.
  2. The covariance matrix has at most four zero eigenvalues because centering removes only one independent direction.
  3. The covariance matrix has at least four zero eigenvalues because the centered data matrix has rank at most four. (correct answer)
  4. The covariance matrix has exactly four positive eigenvalues because the sample size determines its rank uniquely.
Explanation: Whenever you see a PCA question involving a data matrix with more variables than observations, your first instinct should be to think about matrix rank, not just the number of variables. Here's the key insight: start with your centered data matrix X\mathbf{X}, which is n×p=5×8n \times p = 5 \times 8. Centering each column (subtracting the sample mean) imposes one linear constraint — the columns now sum to zero — which reduces the row rank by at most 1. Therefore, rank(X)≤n−1=4\text{rank}(\mathbf{X}) \leq n - 1 = 4. The sample covariance matrix is proportional to X⊤X\mathbf{X}^\top \mathbf{X}, and a fundamental theorem of linear algebra tells us rank(X⊤X)=rank(X)≤4\text{rank}(\mathbf{X}^\top \mathbf{X}) = \text{rank}(\mathbf{X}) \leq 4. Since X⊤X\mathbf{X}^\top \mathbf{X} is an 8×88 \times 8 matrix with rank at most 4, it must have at least 8−4=48 - 4 = 4 zero eigenvalues. This confirms C is guaranteed to be true. A is wrong because it claims exactly three zero eigenvalues using only the difference p−n=3p - n = 3. This ignores the centering constraint and conflates nn with n−1n-1. B is wrong in the opposite direction — it claims centering removes only one direction, leaving at most four zero eigenvalues. But centering reduces available rank to n−1=4n-1=4, guaranteeing at least four zeros, not at most. D is wrong because while nn limits rank, the rank could be less than 4 if columns of X\mathbf{X} are linearly dependent, so you cannot guarantee exactly four positive eigenvalues. Study tip: Memorize this chain — centering an n×pn \times p matrix caps rank at min⁡(n−1,p)\min(n-1, p), which directly bounds how many nonzero eigenvalues the covariance matrix can have. "At least" vs. "exactly" language is almost always a trap.

Question 9

A population covariance matrix has spectral decomposition Σ=Qdiag⁡(5,5,1)QT\Sigma=Q\operatorname{diag}(5,5,1)Q^T, where Q=(q1,q2,q3)Q=(q_1,q_2,q_3) is orthogonal.

Which conclusion about the population principal components is correct?

  1. The first loading is uniquely q1q_1, but its sign can be reversed without changing the component variance.
  2. Any unit vector in span⁡(q1,q2)\operatorname{span}(q_1,q_2) can serve as a first loading, while that two-dimensional principal subspace is unique. (correct answer)
  3. Only q1q_1 and q2q_2 can serve as first loadings, because other rotations would produce correlated component scores.
  4. The first principal subspace is not identifiable, but the individual first and second loading vectors are uniquely determined.
Explanation: When eigenvalues repeat in a spectral decomposition, the geometry of principal components changes fundamentally — and that's exactly what this question tests. The key is distinguishing between the principal subspace and the individual loading vectors. Here, Σ=Q diag(5,5,1)QT\Sigma = Q\,\text{diag}(5,5,1)Q^T has a repeated eigenvalue of 5 for the first two components. This means any unit vector lying in span(q1,q2)\text{span}(q_1, q_2) satisfies Σv=5v\Sigma v = 5v, making it a valid eigenvector with variance 5. There is no mathematical basis for preferring q1q_1 over q2q_2, or any rotation within that plane. However, the two-dimensional subspace itself is uniquely determined — it's the eigenspace corresponding to eigenvalue 5, which is geometrically fixed regardless of which basis you choose for it. This is exactly what B describes, making it correct. A is wrong because it assumes q1q_1 is the unique first loading. With a repeated eigenvalue, the entire eigenspace qualifies — not just the columns of QQ as written. The sign-reversal comment is a red herring that distracts from the real issue of non-uniqueness. C is wrong on two counts: first, any unit vector in span(q1,q2)\text{span}(q_1,q_2) (not just q1q_1 or q2q_2) is valid; second, rotations within the eigenspace preserve uncorrelatedness of component scores, so the correlation argument is false. D reverses the truth. The subspace is identifiable and unique; it's the individual loading vectors within that subspace that are not. Study tip: On any PCA question, immediately check whether eigenvalues repeat. Repeated eigenvalues → the subspace is unique but individual loadings within it are not. Distinct eigenvalues → loadings are unique up to sign.