Singular Value Decomposition (SVD) - Linear Algebra

1. Consider a matrix $A$ with SVD $A = U\Sigma V^T$ where $\Sigma = \text{diag}(6, 4, 2, 0)$. If we form the rank-2 approximation $A_2$ using the first two singular values, what percentage of the Frobenius norm of $A$ is preserved in $A_2$?
   Answer: Approximately 92.3% When you encounter SVD approximation problems, you're dealing with how well lower-rank matrices can represent the original data. The key insight is that the Frobenius norm has a direct relationship with singular values. For any matrix with SVD $A = U\Sigma V^T$, the Frobenius norm squared equals the sum of squared singular values: $||A||_F^2 = \sigma_1^2 + \sigma_2^2 + \sigma_3^2 + \sigma_4^2$. With $\Sigma = \text{diag}(6, 4, 2, 0)$, we get $||A||_F^2 = 36 + 16 + 4 + 0 = 56$. The rank-2 approximation $A_2$ uses only the first two singular values, so $||A_2||_F^2 = 36 + 16 = 52$. The percentage preserved is $\frac{52}{56} = \frac{13}{14} \approx 0.923 = 92.3\%$. Looking at the wrong answers: Choice A (76.9%) might come from incorrectly using $\frac{6+4}{6+4+2+0} = \frac{10}{12}$, which adds singular values instead of squaring them. Choice B (83.3%) could result from the calculation $\frac{10}{12} = 0.833$, also from the linear addition error. Choice C (50%) might stem from thinking half the singular values means half the norm is preserved, ignoring that larger singular values contribute disproportionately. Remember: In SVD approximations, always square the singular values when working with Frobenius norms. The largest singular values capture most of the matrix's "energy," so low-rank approximations often preserve surprisingly high percentages of the original norm.
2. A data matrix $X$ has SVD $X = U\Sigma V^T$ with singular values $\{10, 6, 3, 1\}$. For principal component analysis, if we want to retain at least 90% of the variance, how many principal components should we keep?
   Answer: 2 In PCA, the variance explained by each component is proportional to $\sigma_i^2$. Total variance: $10^2 + 6^2 + 3^2 + 1^2 = 100 + 36 + 9 + 1 = 146$. We need 90% of 146 = 131.4. First component: 100 (68.5%). First two: 100 + 36 = 136 (93.2%). Since 136 > 131.4, we need 2 components. Choice A uses 3 components (unnecessary). Choice C uses all components. Choice D uses only 1 component (insufficient at 68.5%).
3. For a matrix $A$ with singular value decomposition $A = U\Sigma V^T$, suppose the singular values are $\sigma_1 = 8$, $\sigma_2 = 3$, and $\sigma_3 = 0$. If $B = A^T A$, what is the trace of $B$?
   Answer: 73 Since $A = U\Sigma V^T$, we have $A^T = V\Sigma^T U^T$. Therefore, $B = A^T A = V\Sigma^T U^T U\Sigma V^T = V\Sigma^T\Sigma V^T$. The eigenvalues of $B$ are the diagonal entries of $\Sigma^T\Sigma$, which are $\sigma_1^2 = 64$, $\sigma_2^2 = 9$, and $\sigma_3^2 = 0$. The trace equals the sum of eigenvalues: $64 + 9 + 0 = 73$. Choice B uses only $\sigma_1^2$. Choice C uses $\sigma_1 + \sigma_2$. Choice D uses $3 \times \sigma_1$.
4. Consider the matrix $A = \begin{bmatrix} 3 & 1 \\ 4 & 2 \end{bmatrix}$. If the largest singular value of $A$ is approximately 5.46, what is the smallest singular value approximately?
   Answer: 0.37 For a $2 \times 2$ matrix, the product of singular values equals $|\det(A)|$. Here, $\det(A) = 3 \cdot 2 - 1 \cdot 4 = 2$. If $\sigma_1 \approx 5.46$, then $\sigma_1 \sigma_2 = 2$, so $\sigma_2 = \frac{2}{5.46} \approx 0.37$. Choice A uses $\frac{4}{5.46}$. Choice B subtracts: $5.46 - 4$. Choice C assumes the singular values are eigenvalues and uses symmetry incorrectly.
5. In the reduced SVD of an $m \times n$ matrix $A$ with rank $r$, which statement about the orthogonal matrices is correct?
   Answer: $U$ has $r$ orthonormal columns and $V$ has $r$ orthonormal columns When you encounter SVD questions, focus on understanding what "reduced" means and how rank affects matrix dimensions. The reduced SVD is a compact form that eliminates unnecessary zero singular values and their corresponding vectors. In the reduced SVD $A = U\Sigma V^T$, the matrix $A$ has rank $r$, meaning it has exactly $r$ non-zero singular values. The reduced form constructs matrices to match this rank: $U$ becomes $m \times r$, $\Sigma$ becomes $r \times r$, and $V^T$ becomes $r \times n$. Since $U$ has dimensions $m \times r$, it contains $r$ orthonormal columns. Similarly, $V$ has dimensions $n \times r$, so it also contains $r$ orthonormal columns. Choice A describes the full SVD, not the reduced version. In the full SVD, $U$ would indeed have $m$ orthonormal columns and $V$ would have $n$ orthonormal columns, but this includes columns corresponding to zero singular values that are eliminated in the reduced form. Choice C incorrectly focuses on rows rather than columns. While $U$ and $V$ do have $r$ rows each in their transpose forms, the question asks about the orthogonal matrices themselves, where we count orthonormal columns. Choice D suggests the dimensions depend on $\min(m,r)$ and $\min(n,r)$, but since $r$ cannot exceed either $m$ or $n$ (rank is bounded by both dimensions), these expressions simply equal $r$. Study tip: Remember that "reduced" SVD means "rank-sized" - everything scales to match the rank $r$.
6. Let $A$ be a $4 \times 5$ matrix with rank 3. In the singular value decomposition $A = U\Sigma V^T$, what are the dimensions of the matrices $U$, $\Sigma$, and $V$?
   Answer: $U$ is $4 \times 4$, $\Sigma$ is $4 \times 5$, $V$ is $5 \times 5$ In the full SVD, $U$ is always $m \times m$ (here $4 \times 4$), $\Sigma$ is $m \times n$ (here $4 \times 5$), and $V$ is $n \times n$ (here $5 \times 5$). The rank only affects how many singular values are nonzero, not the matrix dimensions. Choice B gives the reduced SVD dimensions. Choice C reverses some dimensions. Choice D incorrectly uses the rank as a matrix dimension.
7. If $A$ is a $3 \times 4$ matrix with singular values $\sigma_1 = 7$, $\sigma_2 = 5$, $\sigma_3 = 0$, what is the dimension of the null space of $A^T$?
   Answer: 1 When you encounter singular values in a linear algebra problem, you're dealing with the Singular Value Decomposition (SVD), which reveals crucial information about a matrix's fundamental subspaces. The key insight is that zero singular values directly tell you about the dimensions of the null spaces. For any $m \times n$ matrix, the SVD gives you $r$ non-zero singular values, where $r$ is the rank of the matrix. Since $A$ has singular values $\sigma_1 = 7$, $\sigma_2 = 5$, and $\sigma_3 = 0$, only two are non-zero, so $\text{rank}(A) = 2$. The dimension of the null space of $A^T$ equals $m - \text{rank}(A)$, where $m$ is the number of rows. Since $A$ is $3 \times 4$, we have $\dim(\text{null}(A^T)) = 3 - 2 = 1$. This makes C correct. Looking at the wrong answers: A) 3 incorrectly assumes the null space dimension equals the number of rows. B) 2 mistakenly uses the rank itself rather than subtracting it from the row count. D) 0 would only be true if $A$ had full row rank (rank 3), which would require all three singular values to be non-zero. Remember this pattern: for an $m \times n$ matrix with $r$ non-zero singular values, $\dim(\text{null}(A^T)) = m - r$ and $\dim(\text{null}(A)) = n - r$. The number of zero singular values always equals the total dimension of both null spaces combined.
8. For a matrix $A$ with SVD $A = U\Sigma V^T$, which expression correctly represents the pseudoinverse $A^+$?
   Answer: $V\Sigma^+ U^T$ where $\Sigma^+$ has $1/\sigma_i$ for nonzero $\sigma_i$ and 0 elsewhere When you encounter pseudoinverse problems, remember that the pseudoinverse extends the concept of matrix inversion to non-square or singular matrices using the Singular Value Decomposition (SVD). The pseudoinverse formula derives directly from the SVD structure. If $A = U\Sigma V^T$, then the pseudoinverse is $A^+ = V\Sigma^+ U^T$. Notice how this "reverses" the SVD: the $V$ and $U$ matrices swap positions and lose their transposes, while $\Sigma^+$ is the pseudoinverse of the diagonal matrix $\Sigma$. For $\Sigma^+$, you take the reciprocal of each nonzero singular value and leave zeros as zeros (since $1/0$ is undefined). This construction ensures $A^+$ exists even when $A$ isn't invertible. Answer C correctly gives $V\Sigma^+ U^T$ with the proper handling of $\Sigma^+$. Answer A incorrectly uses $V^T\Sigma^+ U$, which has the wrong transposes—this would give you something like $(A^T)^+$ instead of $A^+$. Answer B attempts $U\Sigma^{-1} V^T$, but $\Sigma^{-1}$ doesn't exist when $A$ has zero singular values, and the matrix order is wrong anyway. Answer D suggests using standard matrix inversion on $V\Sigma U^T$, but this expression isn't even equal to $A$, and standard inversion fails for non-invertible matrices. Study tip: Remember the pseudoinverse formula as "flip and transpose": from $U\Sigma V^T$ to $V\Sigma^+ U^T$. The matrices swap positions and drop their transposes, while $\Sigma^+$ uses reciprocals of nonzero values only.

Start this quiz