Covariance Matrices & Eigen Decomposition - Statistics Graduate Level

1. Three standardized random variables have pairwise correlations $\operatorname{Corr}(X,Y)=\operatorname{Corr}(Y,Z)=\rho$ and $\operatorname{Corr}(X,Z)=0$, where $\rho\ge 0$. What is the largest value of $\rho$ for which these correlations can define a valid correlation matrix?
   Answer: $\rho=\frac{1}{\sqrt{2}}$ When you encounter a question about valid correlation structures, your first instinct should be to think about positive semi-definiteness. A correlation matrix is valid if and only if it is positive semi-definite (PSD), meaning all its eigenvalues must be non-negative. The correlation matrix for $(X, Y, Z)$ with the given structure is: $R = \begin{pmatrix} 1 & \rho & 0 \\ \rho & 1 & \rho \\ 0 & \rho & 1 \end{pmatrix}$ For $R$ to be PSD, its determinant must be non-negative (a necessary condition that is also sufficient here alongside checking leading minors). Computing: $\det(R) = 1(1 - \rho^2) - \rho(\rho - 0) + 0 = 1 - \rho^2 - \rho^2 = 1 - 2\rho^2$ Setting $\det(R) \geq 0$ gives $\rho^2 \leq \frac{1}{2}$, so $\rho \leq \frac{1}{\sqrt{2}}$. This confirms answer B is correct — the largest valid $\rho$ is exactly $\frac{1}{\sqrt{2}}$. As for the distractors: A ($\rho = \frac{1}{2}$) is a valid correlation but is not the maximum — larger values up to $\frac{1}{\sqrt{2}}$ are also achievable, so it fails the "largest" criterion. C ($\rho = \frac{\sqrt{3}}{2}$) yields $\det(R) = 1 - 2(3/4) = -\frac{1}{2} < 0$, making the matrix invalid. D ($\rho = 1$) would force $X = Y = Z$ almost surely, directly contradicting $\operatorname{Corr}(X,Z) = 0$. Study tip: Whenever a question asks for the feasible range of a correlation parameter, immediately write out the correlation matrix and apply the PSD conditions — start with the determinant, then check all leading principal minors.
2. Which expression gives the first standardized principal-component score in terms of the original variables, together with its variance?
   Answer: The score is $\frac{X_1}{3\sqrt{2}}+\frac{X_2}{2\sqrt{2}}$, and its variance is $\frac{3}{2}$. When PCA is performed on standardized variables, you're working with the correlation matrix, not the original covariance matrix. The standardized variables are $Z_1 = X_1/3$ and $Z_2 = X_2/2$ (dividing by the standard deviations $\sqrt{9}=3$ and $\sqrt{4}=2$). The correlation between them is $\rho = 3/(3 \cdot 2) = 1/2$, giving correlation matrix $R = \begin{pmatrix}1 & 1/2 \\ 1/2 & 1\end{pmatrix}$. The eigenvalues of $R$ are $\lambda_1 = 3/2$ and $\lambda_2 = 1/2$. The first eigenvector (for $\lambda_1 = 3/2$) is $\mathbf{e}_1 = (1/\sqrt{2}, 1/\sqrt{2})^\top$. The first standardized principal component score is therefore $PC_1 = \frac{Z_1}{\sqrt{2}} + \frac{Z_2}{\sqrt{2}} = \frac{X_1}{3\sqrt{2}} + \frac{X_2}{2\sqrt{2}}$, with variance equal to the eigenvalue $\lambda_1 = 3/2$. This confirms A is correct. B is wrong because it uses $X_1$ and $X_2$ directly without standardizing — it treats the original variables as if they already had unit variance. C applies the correct coefficients but uses a minus sign, corresponding to the second eigenvector (for $\lambda_2 = 1/2$), not the first. D swaps the denominators — it divides $X_1$ by $2\sqrt{2}$ and $X_2$ by $3\sqrt{2}$, reversing which standard deviation belongs to which variable. Study tip: Always remember that standardized PCA operates on the correlation matrix $R$, not $\Sigma$. Translate eigenvector scores back to original variables by dividing each $X_i$ by its own standard deviation $\sigma_i$ — mixing up those denominators is the most common trap here.
3. What are the largest eigenvalue and a corresponding unit eigenvector of the covariance matrix after adding the calibration error?
   Answer: The eigenvalue is $9$, with eigenvector $(2q_1+q_2)/\sqrt{5}$. When a rank-1 perturbation is added to a covariance matrix, you need to find the new eigenstructure. The updated covariance matrix is $\Sigma' = \Sigma + 4uu^T$, where $u = (q_1+q_2)/\sqrt{2}$. Since $uu^T = \frac{1}{2}(q_1q_1^T + q_1q_2^T + q_2q_1^T + q_2q_2^T)$, the perturbation mixes only the $q_1$ and $q_2$ subspace — the $q_3$ direction is unaffected, retaining eigenvalue 1. To find the new eigenvalues in the $\{q_1, q_2\}$ subspace, express $\Sigma'$ restricted to that subspace. The original part contributes the diagonal matrix $\text{diag}(6,3)$, and $4uu^T$ contributes $4 \cdot \frac{1}{2}\begin{pmatrix}1&1\\1&1\end{pmatrix} = \begin{pmatrix}2&2\\2&2\end{pmatrix}$. So the restricted matrix is $\begin{pmatrix}8&2\\2&5\end{pmatrix}$. Its eigenvalues satisfy $(8-\lambda)(5-\lambda)-4=0$, giving $\lambda^2 - 13\lambda + 36 = 0$, so $\lambda = 9$ or $\lambda = 4$. For $\lambda=9$: $(8-9)v_1 + 2v_2 = 0 \Rightarrow v_2 = v_1/2$, normalized to $(2q_1+q_2)/\sqrt{5}$. This confirms answer B — eigenvalue 9 with eigenvector $(2q_1+q_2)/\sqrt{5}$. Choice A assumes the perturbation direction $u$ itself is an eigenvector, but rank-1 updates to matrices with unequal existing eigenvalues generally rotate the eigenvectors. Choice C guesses $q_1$ remains an eigenvector, which fails once off-diagonal coupling is introduced. Choice D picks the correct functional form but wrong coefficients — the eigenvector for $\lambda=4$ is $(q_1-2q_2)/\sqrt{5}$, not for the largest eigenvalue. When you see a rank-1 update, always restrict the problem to the affected subspace and solve a small eigenvalue problem there — this avoids dimension-bloat and reveals exactly how eigenvectors rotate.
4. As the relative orientation of their eigenvectors varies, what is the full possible range of the largest eigenvalue of $\Sigma_1+\Sigma_2$?
   Answer: The range is from $5$ through $8$. When two symmetric matrices share the same eigenvalues but have different eigenvectors, their sum's eigenvalues depend critically on how their principal axes are aligned. This is a question about the Weyl inequalities and the geometry of eigenvalue addition. Start by noting that each $\Sigma_i$ has trace $4 + 1 = 5$, so $\Sigma_1 + \Sigma_2$ always has trace $10$. This means the two eigenvalues of the sum always add to $10$, so finding the largest eigenvalue tells you both. When eigenvectors are aligned, both matrices are simultaneously diagonalizable. The largest eigenvalue of the sum is simply $4 + 4 = 8$, and the smallest is $1 + 1 = 2$. This is the maximum possible for the largest eigenvalue. When eigenvectors are orthogonal (rotated $90°$ relative to each other), one matrix "spreads" variance in the direction the other "compresses" it. You can verify explicitly: if $\Sigma_1 = \text{diag}(4,1)$ and $\Sigma_2$ has its large eigenvector along the second axis, then $\Sigma_1 + \Sigma_2 = \text{diag}(5,5)$, giving largest eigenvalue $5$. This is the minimum. By continuity, as orientation varies continuously between these extremes, the largest eigenvalue achieves every value in $[5, 8]$, making D correct. A is wrong because it sets the lower bound at $4$, which is impossible — the sum of two positive definite matrices must have eigenvalues larger than either individual matrix's minimum. B incorrectly claims the eigenvalues are always maximally aligned. C correctly identifies the minimum but wrongly claims it's the only possibility. Remember: for eigenvalue problems involving sums of matrices, trace constraints give you sum of eigenvalues for free, and extreme orientations (parallel vs. orthogonal eigenvectors) give you the range endpoints.
5. For $Z=WX$, which transformation both gives $\operatorname{Cov}(Z)=I$ and makes the $k$th component of $Z$ a rescaled projection onto the $k$th eigenvector?
   Answer: $W=\Lambda^{-1/2}Q^{\mathsf T}$ When whitening a random vector, you need a transformation $W$ such that $\text{Cov}(WX) = W\Sigma W^{\mathsf T} = I$. The spectral decomposition $\Sigma = Q\Lambda Q^{\mathsf T}$ gives you everything you need — but the order of operations matters critically. Choice A, $W = \Lambda^{-1/2}Q^{\mathsf T}$, is correct. Verify it: $W\Sigma W^{\mathsf T} = \Lambda^{-1/2}Q^{\mathsf T}(Q\Lambda Q^{\mathsf T})Q\Lambda^{-1/2} = \Lambda^{-1/2}(Q^{\mathsf T}Q)\Lambda(Q^{\mathsf T}Q)\Lambda^{-1/2} = \Lambda^{-1/2}\Lambda\Lambda^{-1/2} = I$. Crucially, the $k$th component of $Z = WX$ is $\lambda_k^{-1/2}(q_k^{\mathsf T}x)$ — a rescaled projection of $X$ onto the $k$th eigenvector $q_k$. This is precisely the PCA score, normalized to unit variance. Choice B, $W = Q\Lambda^{-1/2}Q^{\mathsf T}$, is the symmetric whitening (Mahalanobis) transform. While $\text{Cov}(WX) = I$ holds here too, the components of $Z$ are not projections onto individual eigenvectors — they're linear combinations of all eigenvectors, failing the second requirement. Choice C, $W = Q^{\mathsf T}\Lambda^{-1/2}$, reverses the order of $Q^{\mathsf T}$ and $\Lambda^{-1/2}$. Computing $W\Sigma W^{\mathsf T}$ yields $Q^{\mathsf T}\Lambda^{-1/2}Q\Lambda Q^{\mathsf T}\Lambda^{-1/2}Q$, which does not simplify to $I$ because $\Lambda^{-1/2}$ and $Q$ don't commute. Choice D, $W = \Lambda^{1/2}Q^{\mathsf T}$, uses $\Lambda^{1/2}$ instead of $\Lambda^{-1/2}$, which would amplify variance rather than normalize it, giving $\text{Cov}(Z) = \Lambda^2 \neq I$. Your strategy: always verify whitening candidates by computing $W\Sigma W^{\mathsf T}$ explicitly, and separately check whether components correspond to individual eigenvector projections — both conditions must hold simultaneously.
6. Along the eigenvector associated with the smallest eigenvalue of $\Sigma$, what is the ratio of the sampling variance of the ridge coefficient to that of the ordinary least-squares coefficient?
   Answer: The variance ratio is $1/4$. When comparing ridge and OLS variance along a specific eigendirection, the key is to work in the eigenspace of $X^\mathsf{T}X$ and track how each estimator scales the signal. For OLS, the covariance matrix is $\sigma^2(X^\mathsf{T}X)^{-1}$. Along an eigenvector with eigenvalue $\phi$ of $X^\mathsf{T}X$, the OLS variance is $\sigma^2/\phi$. For ridge, the covariance is $\sigma^2(X^\mathsf{T}X+\lambda I)^{-1}(X^\mathsf{T}X)(X^\mathsf{T}X+\lambda I)^{-1}$, giving variance $\sigma^2\phi/(\phi+\lambda)^2$ along that same eigenvector. The variance ratio (ridge to OLS) is therefore: $\frac{\sigma^2\phi/(\phi+\lambda)^2}{\sigma^2/\phi} = \frac{\phi^2}{(\phi+\lambda)^2}$ Now plug in the numbers. The smallest eigenvalue of $\Sigma$ is $1/4$, and $X^\mathsf{T}X = n\Sigma$, so the corresponding eigenvalue of $X^\mathsf{T}X$ is $\phi = n/4$. With $\lambda = n/4$: $\frac{\phi^2}{(\phi+\lambda)^2} = \frac{(n/4)^2}{(n/4+n/4)^2} = \frac{(n/4)^2}{(n/2)^2} = \frac{1}{4}$ This confirms C is correct. Choice A ($1/8$) has no algebraic basis here — it might tempt you if you mistakenly take a cube rather than square. Choice B ($1/2$) is the ratio $\phi/(\phi+\lambda)$, which is only one factor of the correct squared expression — a common half-finished calculation error. Choice D ($1/16$) results from squaring the wrong base ratio, perhaps computing $(1/4)^2$ directly without correctly forming $\phi/(\phi+\lambda)$. Study tip: Always derive the ridge variance formula from scratch using the sandwich form — memorizing only the OLS formula and forgetting the extra $\phi$ in the numerator is the most frequent trap on problems like this.
7. How many zero eigenvalues does the sample covariance matrix have?
   Answer: It has exactly $4$ zero eigenvalues because centering leaves rank $4$. Whenever you see a question about eigenvalues of a sample covariance matrix, your first instinct should be to think about rank, not about the divisor or the population's properties. Here's the key chain of reasoning. You start with $n = 5$ observations in $p = 8$ dimensions. Stack these into a data matrix $X$ of size $5 \times 8$. Before centering, this matrix has rank at most $\min(5, 8) = 5$. When you subtract the sample mean, you project the rows onto a subspace orthogonal to the all-ones vector, which reduces the row rank by exactly 1. So the centered data matrix has rank at most $4$. The sample covariance matrix $S = \frac{1}{n-1} X_c^\top X_c$, where $X_c$ is the centered matrix, is an $8 \times 8$ matrix with rank at most $4$. By the rank-nullity theorem, a rank-4 matrix in $\mathbb{R}^8$ has a null space of dimension $8 - 4 = 4$, meaning exactly 4 zero eigenvalues. This confirms answer B. Choice A incorrectly uses the rank of the uncentered data matrix (rank 5), forgetting that centering costs one additional rank. Choice C confuses the $n-1$ denominator — that's just a scalar rescaling and has absolutely no effect on the number of zero eigenvalues. Choice D is a tempting trap: continuity of the population distribution guarantees the population covariance matrix is positive definite, but the sample covariance matrix is constrained by the geometry of finitely many observations, not the population. Remember: for sample covariance matrices, always think $\text{rank} = \min(n-1, p)$, and zero eigenvalues $= p - \text{rank}$.
8. For the generalized eigenproblem $B a=\lambda W a$, which pair gives the nonzero generalized eigenvalue and a corresponding direction?
   Answer: The eigenvalue is $2$, with direction proportional to $(1/2,1)^{\mathsf T}$. When you encounter a generalized eigenproblem $Ba = \lambda Wa$ in LDA, the key insight is that you can convert it to a standard eigenproblem by computing $W^{-1}B$, then find eigenvalues and eigenvectors normally. Here, $B = vv^\mathsf{T} = \begin{pmatrix}4&2\\2&1\end{pmatrix}$ and $W^{-1} = \text{diag}(1/4, 1)$, so: $W^{-1}B = \begin{pmatrix}1/4&0\\0&1\end{pmatrix}\begin{pmatrix}4&2\\2&1\end{pmatrix} = \begin{pmatrix}1&1/2\\2&1\end{pmatrix}$ Since $B$ is rank-1, there is exactly one nonzero eigenvalue. The trace gives $\lambda = 1 + 1 = 2$. For the eigenvector, solve $(W^{-1}B - 2I)a = 0$: the system gives $-a_1 + \frac{1}{2}a_2 = 0$, so $a_1 = \frac{1}{2}a_2$, yielding direction $(1/2, 1)^\mathsf{T}$. This confirms answer A is correct. Answer B is tempting because $(2,1)^\mathsf{T}$ is the mean-difference vector $v$ itself, but the LDA direction is $W^{-1}v$, not $v$. Plugging into the eigenproblem, $W^{-1}B(2,1)^\mathsf{T} = (2,5)^\mathsf{T}$, which is not a scalar multiple of $(2,1)^\mathsf{T}$, so it fails. Answer C uses direction $(1,1/2)^\mathsf{T}$ with eigenvalue 1 — neither checks out against $W^{-1}B$. Answer D offers a scaled version of $v$ with a fractional eigenvalue, which is simply numerically inconsistent with the trace calculation. A useful habit: always verify a candidate eigenpair by checking $W^{-1}Ba = \lambda a$ directly — it takes seconds and eliminates traps built around plausible-looking but incorrect directions.
9. What are the eigenvalues of the conditional covariance matrix of $(X_1,X_2)^{\mathsf T}$ given $Y$?
   Answer: The conditional eigenvalues are $2$ and $1$. When you see a question about conditional distributions in a jointly Gaussian setting, your first instinct should be the standard Gaussian conditioning formula. For a jointly Gaussian vector, the conditional covariance of $(X_1, X_2)^\mathsf{T}$ given $Y$ is: $\Sigma_{X|Y} = \Sigma_{XX} - \Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YX}$ Here, $\Sigma_{XX} = \begin{pmatrix}2&1\\1&2\end{pmatrix}$, $\Sigma_{XY} = \begin{pmatrix}1\\1\end{pmatrix}$, and $\Sigma_{YY} = 2$. Plugging in: $\Sigma_{X|Y} = \begin{pmatrix}2&1\\1&2\end{pmatrix} - \frac{1}{2}\begin{pmatrix}1\\1\end{pmatrix}\begin{pmatrix}1&1\end{pmatrix} = \begin{pmatrix}2&1\\1&2\end{pmatrix} - \begin{pmatrix}1/2&1/2\\1/2&1/2\end{pmatrix} = \begin{pmatrix}3/2&1/2\\1/2&3/2\end{pmatrix}$ The eigenvalues of $\begin{pmatrix}3/2&1/2\\1/2&3/2\end{pmatrix}$ are $3/2 + 1/2 = 2$ and $3/2 - 1/2 = 1$, confirming answer D. Choice A (eigenvalues 3 and 1) corresponds to the eigenvalues of the unconditional $\Sigma_{XX}$, forgetting to subtract the correction term entirely. Choice B (eigenvalues 3/2 and 1/2) is a subtle trap: these are the diagonal entries of $\Sigma_{X|Y}$, which students sometimes confuse for eigenvalues — a common mistake when the matrix isn't diagonal. Choice C (eigenvalues 2 and 0) likely arises from incorrectly computing the correction term, perhaps treating $\Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YX}$ as the full original matrix. As a study tip: always distinguish between diagonal entries and eigenvalues. For a symmetric $2\times2$ matrix $\begin{pmatrix}a&b\\b&a\end{pmatrix}$, eigenvalues are $a+b$ and $a-b$ — memorize this pattern to avoid the trap in choice B.
10. Which statement correctly describes the leading principal-component eigenspace?
    Answer: It is the three-dimensional contrast space orthogonal to $\mathbf{1}$, with eigenvalue $1.2$. When you encounter a question about PCA on an exchangeable correlation matrix, your first move should be to find the eigenvalues analytically using the known structure of $R = (1-\rho)I_4 + \rho\mathbf{1}\mathbf{1}^\mathsf{T}$. This matrix has exactly two distinct eigenvalues. The vector $\mathbf{1}$ is an eigenvector with eigenvalue $\lambda_1 = (1-\rho) + 4\rho = 1 + 3\rho$. Any vector orthogonal to $\mathbf{1}$ (a "contrast vector") is an eigenvector with eigenvalue $\lambda_2 = 1 - \rho$. Plugging in $\rho = -0.2$: $\lambda_1 = 1 + 3(-0.2) = 0.4$ $\lambda_2 = 1 - (-0.2) = 1.2$ Since $1.2 > 0.4$, the leading principal component corresponds to $\lambda_2 = 1.2$, whose eigenspace is the three-dimensional space of all vectors orthogonal to $\mathbf{1}$ — the contrast space. That confirms C is correct. Answer A gets the eigenspace wrong — it claims $\mathbf{1}$ spans the leading component — and assigns it $1.2$, which is actually the contrast eigenvalue. Answer B correctly identifies that $\mathbf{1}$ has eigenvalue $0.4$, but calling it the leading component is the critical error; it's actually the smallest eigenvalue here. Answer D correctly identifies the contrast space but assigns it the wrong eigenvalue, $0.4$ instead of $1.2$. The key study tip: with negative $\rho$, the roles of $\mathbf{1}$ and the contrast space flip compared to the positive-$\rho$ case. Always compute both eigenvalues before deciding which dominates.

Start this quiz