Least Squares Solutions - Linear Algebra

Consider the matrix equation $A\mathbf{x} = \mathbf{b}$ where $A$ has linearly independent columns. If $P$ is the orthogonal projection matrix onto the column space of $A$ , and $\mathbf{x}^*$ is the least squares solution, which expression correctly represents the projection of $\mathbf{b}$ onto $\text{Col}(A)$ ?
Answer: $A\mathbf{x}^* = A(A^T A)^{-1}A^T \mathbf{b}$ When you encounter least squares problems, you're dealing with finding the best approximation when the system $A\mathbf{x} = \mathbf{b}$ has no exact solution. The key insight is that the least squares solution $\mathbf{x}^*$ produces $A\mathbf{x}^*$ , which is the orthogonal projection of $\mathbf{b}$ onto the column space of $A$ . The projection of $\mathbf{b}$ onto $\text{Col}(A)$ is given by $P\mathbf{b} = A(A^TA)^{-1}A^T\mathbf{b}$ , where $P$ is the orthogonal projection matrix. Since $A$ has linearly independent columns, $A^TA$ is invertible, making this formula valid. The least squares solution is $\mathbf{x}^* = (A^TA)^{-1}A^T\mathbf{b}$ , so $A\mathbf{x}^* = A(A^TA)^{-1}A^T\mathbf{b}$ , which is exactly the projection formula. Option A incorrectly claims $AA^T\mathbf{b} = A(A^TA)^{-1}A^T\mathbf{b}$ . The matrix $AA^T$ projects onto the column space of $A$ only when $A$ has orthonormal columns, which isn't given here. Option B confuses the least squares solution itself with the projection. The expression $(A^TA)^{-1}A^T\mathbf{b}$ is $\mathbf{x}^*$ , not the projection of $\mathbf{b}$ . The claimed equality with $A^T(AA^T)^{-1}\mathbf{b}$ is also false. Option C incorrectly sets $A^TA\mathbf{x}^* = (A^TA)^{-1}A^T\mathbf{b}$ . Since $\mathbf{x}^* = (A^TA)^{-1}A^T\mathbf{b}$ , the left side would equal $A^T\mathbf{b}$ , which generally differs from the right side. Remember: in least squares problems, $A\mathbf{x}^*$ always represents the projection of $\mathbf{b}$ onto the column space of $A$ .
Under which condition is the least squares solution $\hat{x}$ to the system $Ax = b$ guaranteed to be unique?
Answer: The columns of $A$ are linearly independent. The least squares solution $\hat{x}$ is found by solving the normal equations $A^T A \hat{x} = A^T b$ . This system has a unique solution for $\hat{x}$ if and only if the matrix $A^T A$ is invertible. The matrix $A^T A$ is invertible if and only if the columns of the original matrix $A$ are linearly independent. (B) is incorrect because in least-squares problems, $A$ is typically a non-square matrix and thus cannot be invertible. (C) is a common characteristic of overdetermined systems but does not guarantee linearly independent columns. For example, one column could be a multiple of another. (D) implies that an exact solution exists ( $||A\hat{x}-b||=0$ ), but it does not guarantee the solution is unique; if columns of $A$ are dependent, there could be infinitely many exact solutions.
For a system $Ax = b$ where $A$ has linearly independent columns, the projection of $b$ onto the column space of $A$ is given by $p = Pb$ for some projection matrix $P$ . Which of the following correctly defines $P$ ?
Answer: $P = A(A^T A)^{-1} A^T$ When you encounter projection problems in linear algebra, you're dealing with finding the closest point in a subspace to a given vector. Here, we need the projection matrix that maps any vector $b$ to its projection onto the column space of $A$ . The correct projection matrix is $P = A(A^T A)^{-1} A^T$ . Here's why: when we project $b$ onto the column space of $A$ , we're finding the vector $p$ in that subspace closest to $b$ . This means $p = A\hat{x}$ for some coefficient vector $\hat{x}$ , and the error $(b - p)$ must be perpendicular to the column space. This orthogonality condition gives us $A^T(b - A\hat{x}) = 0$ , which leads to $\hat{x} = (A^T A)^{-1} A^T b$ . Therefore, $p = A\hat{x} = A(A^T A)^{-1} A^T b$ , confirming that $P = A(A^T A)^{-1} A^T$ . Option A gives you $(A^T A)^{-1} A^T$ , which is just the coefficient vector $\hat{x}$ , not the projection itself. Option C, $A A^T$ , only works when $A$ is square, and even then it's not generally a projection matrix. Option D, $A^T A$ , transforms vectors in the domain space rather than projecting onto the column space. Remember this key pattern: projection matrices always have the form $A(A^T A)^{-1} A^T$ when $A$ has linearly independent columns. The middle term $(A^T A)^{-1} A^T$ finds coefficients, while the outer $A$ converts back to the projected vector.
For an overdetermined system $A\mathbf{x} = \mathbf{b}$ where $A$ is $m \times n$ with $m > n$ , the residual vector for the least squares solution $\mathbf{x}^*$ is $\mathbf{r} = \mathbf{b} - A\mathbf{x}^*$ . Which of the following statements about $\mathbf{r}$ is necessarily true?
Answer: $\mathbf{r}$ is orthogonal to every vector in the column space of $A$ and has minimum possible length among all residual vectors The least squares solution minimizes $\|\mathbf{b} - A\mathbf{x}\|^2$ . The residual $\mathbf{r} = \mathbf{b} - A\mathbf{x}^*$ is orthogonal to the column space of $A$ (this is the geometric interpretation of the normal equation $A^T(\mathbf{b} - A\mathbf{x}^*) = \mathbf{0}$ ), and by definition has minimum length among all possible residuals. Choice B incorrectly refers to the null space instead of column space. Choice C confuses row space with column space. Choice D correctly identifies orthogonality to the column space but incorrectly claims the residual lies in the left null space, which would only be true if $\mathbf{r} = \mathbf{0}$ .
What is the orthogonal projection of the vector $b = \begin{pmatrix} 2 \\ 5 \\ 1 \end{pmatrix}$ onto the subspace spanned by the columns of the matrix $A = \begin{pmatrix} 1 & 1 \\ 2 & -1 \\ 1 & 1 \end{pmatrix}$ ?
Answer: $\begin{pmatrix} 1.5 \\ 5 \\ 1.5 \end{pmatrix}$ When you encounter orthogonal projection problems, you're finding the "shadow" of a vector onto a subspace - the closest point in that subspace to your original vector. The formula is $\text{proj}_A(b) = A(A^T A)^{-1}A^T b$ . Let's work through this systematically. First, compute $A^T A$ : $A^T A = \begin{pmatrix} 1 & 2 & 1 \\ 1 & -1 & 1 \end{pmatrix} \begin{pmatrix} 1 & 1 \\ 2 & -1 \\ 1 & 1 \end{pmatrix} = \begin{pmatrix} 6 & 0 \\ 0 & 3 \end{pmatrix}$ The inverse is straightforward: $(A^T A)^{-1} = \begin{pmatrix} 1/6 & 0 \\ 0 & 1/3 \end{pmatrix}$ Next, calculate $A^T b$ : $A^T b = \begin{pmatrix} 1 & 2 & 1 \\ 1 & -1 & 1 \end{pmatrix} \begin{pmatrix} 2 \\ 5 \\ 1 \end{pmatrix} = \begin{pmatrix} 9 \\ -2 \end{pmatrix}$ Now: $(A^T A)^{-1}A^T b = \begin{pmatrix} 1/6 & 0 \\ 0 & 1/3 \end{pmatrix} \begin{pmatrix} 9 \\ -2 \end{pmatrix} = \begin{pmatrix} 3/2 \\ -2/3 \end{pmatrix}$ Finally: $A(A^T A)^{-1}A^T b = \begin{pmatrix} 1 & 1 \\ 2 & -1 \\ 1 & 1 \end{pmatrix} \begin{pmatrix} 3/2 \\ -2/3 \end{pmatrix} = \begin{pmatrix} 1.5 \\ 5 \\ 1.5 \end{pmatrix}$ This confirms answer D. Answer A uses incorrect arithmetic in the matrix multiplications. Answer B would only be correct if $b$ were already in the subspace (which it isn't). Answer C appears to involve calculation errors in the inverse or final multiplication steps. Study tip: Always verify that $A^T A$ is invertible before applying the projection formula, and double-check your matrix arithmetic - small errors compound quickly in multi-step calculations.
Consider two overdetermined systems: System 1 has matrix $A_1$ and right-hand side $\mathbf{b}_1$ , while System 2 has matrix $A_2 = QA_1$ and right-hand side $\mathbf{b}_2 = Q\mathbf{b}_1$ , where $Q$ is an orthogonal matrix. If $\mathbf{x}_1^*$ and $\mathbf{x}_2^*$ are the respective least squares solutions, which relationship holds?
Answer: $\mathbf{x}_2^* = \mathbf{x}_1^*$ and the residual norms satisfy $\|\mathbf{b}_2 - A_2\mathbf{x}_2^*\| = \|\mathbf{b}_1 - A_1\mathbf{x}_1^*\|$ When you encounter problems involving orthogonal transformations of least squares systems, the key insight is that orthogonal matrices preserve geometric relationships while potentially changing coordinates. Let's work through what happens systematically. The least squares solution satisfies the normal equations: $A_1^T A_1 \mathbf{x}_1^* = A_1^T \mathbf{b}_1$ for System 1. For System 2 with $A_2 = QA_1$ and $\mathbf{b}_2 = Q\mathbf{b}_1$ , the normal equations become: $A_2^T A_2 \mathbf{x}_2^* = A_2^T \mathbf{b}_2$ $(QA_1)^T (QA_1) \mathbf{x}_2^* = (QA_1)^T Q\mathbf{b}_1$ $A_1^T Q^T Q A_1 \mathbf{x}_2^* = A_1^T Q^T Q \mathbf{b}_1$ Since $Q$ is orthogonal, $Q^T Q = I$ , so this simplifies to $A_1^T A_1 \mathbf{x}_2^* = A_1^T \mathbf{b}_1$ . This is identical to System 1's normal equations, proving $\mathbf{x}_2^* = \mathbf{x}_1^*$ . For residual norms, since orthogonal matrices preserve lengths: $\|\mathbf{b}_2 - A_2\mathbf{x}_2^*\| = \|Q\mathbf{b}_1 - QA_1\mathbf{x}_1^*\| = \|Q(\mathbf{b}_1 - A_1\mathbf{x}_1^*)\| = \|\mathbf{b}_1 - A_1\mathbf{x}_1^*\|$ . Answer A incorrectly suggests the solution changes by $Q^T$ . Answer B incorrectly claims the solution transforms by $Q$ and misunderstands that $\|Q\| = 1$ for orthogonal matrices. Answer D correctly identifies equal solutions but incorrectly scales the residual norm by $\|Q\|^2$ . Remember: orthogonal transformations preserve the essential geometry of least squares problems—both the solution and the quality of fit remain unchanged.
Consider the overdetermined linear system $Ax = b$ where $A = \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{pmatrix}$ and $b = \begin{pmatrix} 1 \\ 2 \\ 0 \end{pmatrix}$ . To find the least squares solution $\hat{x}$ , which of the following normal equations must be solved?
Answer: $\begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix} \hat{x} = \begin{pmatrix} 1 \\ 2 \end{pmatrix}$ The least squares solution $\hat{x}$ is found by solving the normal equations $A^T A \hat{x} = A^T b$ . First, we compute $A^T A$ and $A^T b$ . $A^T = \begin{pmatrix} 1 & 0 & 1 \\ 0 & 1 & 1 \end{pmatrix}$ . $A^T A = \begin{pmatrix} 1 & 0 & 1 \\ 0 & 1 & 1 \end{pmatrix} \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{pmatrix} = \begin{pmatrix} 1(1)+0(0)+1(1) & 1(0)+0(1)+1(1) \\ 0(1)+1(0)+1(1) & 0(0)+1(1)+1(1) \end{pmatrix} = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}$ . $A^T b = \begin{pmatrix} 1 & 0 & 1 \\ 0 & 1 & 1 \end{pmatrix} \begin{pmatrix} 1 \\ 2 \\ 0 \end{pmatrix} = \begin{pmatrix} 1(1)+0(2)+1(0) \\ 0(1)+1(2)+1(0) \end{pmatrix} = \begin{pmatrix} 1 \\ 2 \end{pmatrix}$ . Thus, the normal equation is $\begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix} \hat{x} = \begin{pmatrix} 1 \\ 2 \end{pmatrix}$ . (B) is incorrect because it uses the equation $A^T A \hat{x} = b$ , failing to multiply $b$ by $A^T$ . (C) is incorrect because it represents $A \hat{x} = A^T b$ . (D) is incorrect because it represents $A^T \hat{x} = b$ .
Let the least squares problem for $Ax = b$ have a solution $\hat{x}$ . If the null space of $A$ , $\text{Nul}(A)$ , contains non-zero vectors, what can be concluded about the set of all least squares solutions?
Answer: The problem has infinitely many least squares solutions. When you encounter least squares problems involving the null space of matrix $A$ , you're dealing with questions about solution uniqueness. The key insight is understanding how the null space affects the solution set structure. The least squares solution $\hat{x}$ satisfies the normal equation $A^T A \hat{x} = A^T b$ . If $\text{Nul}(A)$ contains non-zero vectors, then $A$ doesn't have full column rank. This means $A^T A$ is also not invertible, since $\text{Nul}(A^T A) = \text{Nul}(A)$ . Here's the crucial point: if $\hat{x}$ is one least squares solution and $v$ is any vector in $\text{Nul}(A)$ , then $\hat{x} + v$ is also a least squares solution. This is because $A(\hat{x} + v) = A\hat{x} + Av = A\hat{x} + 0 = A\hat{x}$ , so the residual $||A(\hat{x} + v) - b||$ remains the same. Since $\text{Nul}(A)$ contains infinitely many vectors (it's a subspace), there are infinitely many solutions. Option A is wrong because the problem states a solution $\hat{x}$ exists. Option C contradicts our analysis—uniqueness requires $\text{Nul}(A) = \{0\}$ . Option D is incorrect because the zero vector is rarely the least squares solution unless $b$ is in the column space of $A$ and passes through the origin. Study tip: Remember that in least squares problems, solution uniqueness depends entirely on whether $A$ has full column rank. If $\text{Nul}(A) \neq \{0\}$ , expect infinitely many solutions forming an affine subspace.
For the overdetermined linear system $A\mathbf{x} = \mathbf{b}$ where $A$ is $m \times n$ with $m > n$ and $\text{rank}(A) = n$ , consider the augmented matrix $[A | \mathbf{b}]$ . If the least squares residual has norm $r > 0$ , which statement about the rank of $[A | \mathbf{b}]$ is correct?
Answer: $\text{rank}([A | \mathbf{b}]) = n + 1$ because $\mathbf{b}$ does not lie in the column space of $A$ Since the least squares residual has norm $r > 0$ , this means the system $A\mathbf{x} = \mathbf{b}$ has no exact solution, which occurs precisely when $\mathbf{b}$ is not in the column space of $A$ . Since $A$ has rank $n$ and $\mathbf{b} \notin \text{Col}(A)$ , adding $\mathbf{b}$ as an additional column increases the dimension of the column space by 1. Therefore, $\text{rank}([A | \mathbf{b}]) = n + 1$ . Choice A would be correct if $r = 0$ , but contradicts the given condition. Choice C is incorrect because the rank depends on whether $\mathbf{b}$ is in the column space. Choice D confuses the number of rows with the rank; the rank cannot exceed $n+1$ since there are only $n+1$ columns.
Let $\hat{x}$ be the least squares solution to $Ax = b$ , and let the residual vector be $r = b - A\hat{x}$ . Which of the following statements about $r$ is necessarily true?
Answer: The vector $r$ is orthogonal to the column space of $A$ . When you encounter least squares problems, focus on the fundamental geometric relationship: the residual vector must be orthogonal to the space you're projecting onto. The least squares solution $\hat{x}$ minimizes $||b - Ax||^2$ , which geometrically means we're finding the point $A\hat{x}$ in the column space of $A$ that's closest to $b$ . This creates a projection of $b$ onto the column space. The residual $r = b - A\hat{x}$ represents the "leftover" part of $b$ that couldn't be expressed as a linear combination of $A$ 's columns. By the fundamental theorem of projections, this residual must be orthogonal to everything in the column space of $A$ . This orthogonality condition is what makes the projection unique and optimal. Therefore, answer D is correct. Now for the incorrect options: A claims $||r|| = ||b||$ , which only occurs when $b$ is completely orthogonal to the column space—a special case, not generally true. B suggests $r$ is parallel to $b$ , which again only happens in that same special orthogonal case. C states $r$ lies in the row space of $A$ , but the row space and column space are generally different subspaces, and there's no reason for the residual to align with the row space. Study tip: Remember that least squares is fundamentally about orthogonal projection. The residual is always orthogonal to the space you're projecting onto (the column space). This orthogonality principle appears throughout linear algebra, so master it early.
What is the least squares solution $\hat{x}$ to the inconsistent system $Ax = b$ where $A = \begin{pmatrix} 1 \\ 2 \\ -1 \end{pmatrix}$ and $b = \begin{pmatrix} 1 \\ 1 \\ 1 \end{pmatrix}$ ?
Answer: $\hat{x} = \begin{pmatrix} 1/3 \end{pmatrix}$ To find the least squares solution, we solve the normal equation $A^T A \hat{x} = A^T b$ . $A^T = \begin{pmatrix} 1 & 2 & -1 \end{pmatrix}$ . $A^T A = \begin{pmatrix} 1 & 2 & -1 \end{pmatrix} \begin{pmatrix} 1 \\ 2 \\ -1 \end{pmatrix} = (1^2 + 2^2 + (-1)^2) = (1+4+1) = (6)$ . $A^T b = \begin{pmatrix} 1 & 2 & -1 \end{pmatrix} \begin{pmatrix} 1 \\ 1 \\ 1 \end{pmatrix} = (1(1) + 2(1) - 1(1)) = (1+2-1) = (2)$ . The normal equation is $(6)\hat{x} = (2)$ , which gives $\hat{x} = \begin{pmatrix} 2/6 \end{pmatrix} = \begin{pmatrix} 1/3 \end{pmatrix}$ . (B) results from a sign error in calculating $A^T A$ as $1+4-1=4$ , leading to $4\hat{x}=2$ . (C) results from a sign error in calculating $A^T b$ as $1+2+1=4$ , leading to $6\hat{x}=4$ . (D) represents the value of $A^T b$ , not $\hat{x}$ .
A scientist models an experiment with the equation $y(t) = at + bt^2$ . She collects three data points $(t, y)$ : $(1, 2)$ , $(2, 9)$ , and $(3, 20)$ . Which matrix equation represents the normal equations $A^T A \hat{x} = A^T b$ for finding the least-squares estimate of the parameters $\hat{x} = \begin{pmatrix} a \\ b \end{pmatrix}$ ?
Answer: $\begin{pmatrix} 14 & 36 \\ 36 & 98 \end{pmatrix} \begin{pmatrix} a \\ b \end{pmatrix} = \begin{pmatrix} 80 \\ 218 \end{pmatrix}$ When you encounter a least-squares problem, you're setting up the matrix equation $A\hat{x} = b$ where $A$ contains your input data, $\hat{x}$ holds the unknown parameters, and $b$ contains your observed outputs. The normal equations $A^T A \hat{x} = A^T b$ give you the best-fit solution. For the model $y(t) = at + bt^2$ , your matrix $A$ has columns for each parameter's coefficients. With data points $(1,2)$ , $(2,9)$ , and $(3,20)$ , you get: $A = \begin{pmatrix} 1 & 1 \\ 2 & 4 \\ 3 & 9 \end{pmatrix}, \quad b = \begin{pmatrix} 2 \\ 9 \\ 20 \end{pmatrix}$ Computing $A^T A$ : $A^T A = \begin{pmatrix} 1 & 2 & 3 \\ 1 & 4 & 9 \end{pmatrix} \begin{pmatrix} 1 & 1 \\ 2 & 4 \\ 3 & 9 \end{pmatrix} = \begin{pmatrix} 14 & 36 \\ 36 & 98 \end{pmatrix}$ Computing $A^T b$ : $A^T b = \begin{pmatrix} 1 & 2 & 3 \\ 1 & 4 & 9 \end{pmatrix} \begin{pmatrix} 2 \\ 9 \\ 20 \end{pmatrix} = \begin{pmatrix} 80 \\ 218 \end{pmatrix}$ This confirms answer B is correct. A uses incorrect dimensions and values. C has a singular matrix (identical rows), which can't represent proper normal equations. D shows the original system $A\hat{x} = b$ , not the normal equations. Study tip: Always remember that normal equations transform your original system by multiplying both sides by $A^T$ . The resulting $A^T A$ matrix will be square and symmetric, helping you verify your setup.
Let $\hat{x}$ be the least squares solution to an inconsistent system $Ax = b$ . Which statement best describes the geometric relationship between the vector $p = A\hat{x}$ and the vector $b$ ?
Answer: The vector $p$ is the orthogonal projection of $b$ onto the column space of $A$ . When you encounter least squares problems, you're dealing with inconsistent systems where no exact solution exists, so we find the "best" approximate solution by minimizing the error. The key insight is understanding what $p = A\hat{x}$ represents geometrically. Since $\hat{x}$ is the least squares solution, it minimizes $||A\hat{x} - b||^2$ . This minimization occurs when the error vector $b - A\hat{x}$ is orthogonal to the column space of $A$ . In other words, $b - p$ is perpendicular to every column of $A$ . This geometric condition means that $p = A\hat{x}$ is the point in the column space of $A$ that lies closest to $b$ . By definition, this makes $p$ the orthogonal projection of $b$ onto the column space of $A$ . Option A is incorrect because $p$ and $b$ are generally not orthogonal to each other - rather, the error vector $b - p$ is orthogonal to the column space. Option B confuses column space with row space; the projection occurs onto the column space (the span of $A$ 's columns), not the row space. Option D misunderstands the fundamental nature of inconsistent systems - even though $\hat{x}$ minimizes error, $p$ cannot equal $b$ because $b$ lies outside the column space of $A$ . Remember this pattern: in least squares problems, $A\hat{x}$ always represents the orthogonal projection of $b$ onto the column space of $A$ . The column space connection is crucial since $A\hat{x}$ must be a linear combination of $A$ 's columns.
Let $A$ be an $m \times n$ matrix with $m > n$ . Which of the following statements about the Gram matrix $G = A^T A$ is false?
Answer: $G$ is always invertible. The statement that $G = A^T A$ is always invertible is false. The matrix $G$ is invertible if and only if the columns of $A$ are linearly independent. If the columns of $A$ are linearly dependent, then $A^T A$ will be singular (not invertible). (A) is true because $(A^T A)^T = A^T (A^T)^T = A^T A$ . (B) is true because if $A$ is $m \times n$ , then $A^T$ is $n \times m$ , and their product $A^T A$ is an $n \times n$ square matrix. (D) is true because for any vector $v$ , $v^T G v = v^T A^T A v = (Av)^T(Av) = ||Av||^2 \ge 0$ , which means $G$ is positive semidefinite, and a property of positive semidefinite matrices is that all their eigenvalues are non-negative.
For the system $Ax = b$ with $A = \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{pmatrix}$ and $b = \begin{pmatrix} 6 \\ 0 \\ 0 \end{pmatrix}$ , what is the least squares error, defined as $||A\hat{x} - b||$ ?
Answer: $2\sqrt{3}$ When you encounter a least squares problem, you're dealing with an overdetermined system (more equations than unknowns) that likely has no exact solution. The goal is to find the best approximate solution $\hat{x}$ that minimizes the error $||A\hat{x} - b||$ . To solve this, you need the normal equation: $A^T A \hat{x} = A^T b$ . First, calculate $A^T A$ : $A^T A = \begin{pmatrix} 1 & 0 & 1 \\ 0 & 1 & 1 \end{pmatrix} \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{pmatrix} = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}$ Then $A^T b = \begin{pmatrix} 1 & 0 & 1 \\ 0 & 1 & 1 \end{pmatrix} \begin{pmatrix} 6 \\ 0 \\ 0 \end{pmatrix} = \begin{pmatrix} 6 \\ 0 \end{pmatrix}$ Solving $(A^T A)\hat{x} = A^T b$ : $\begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix} \hat{x} = \begin{pmatrix} 6 \\ 0 \end{pmatrix}$ This gives $\hat{x} = \begin{pmatrix} 4 \\ -2 \end{pmatrix}$ . Now calculate the error: $A\hat{x} - b = \begin{pmatrix} 4 \\ -2 \\ 2 \end{pmatrix} - \begin{pmatrix} 6 \\ 0 \\ 0 \end{pmatrix} = \begin{pmatrix} -2 \\ -2 \\ 2 \end{pmatrix}$ The least squares error is $||A\hat{x} - b|| = \sqrt{(-2)^2 + (-2)^2 + 2^2} = \sqrt{12} = 2\sqrt{3}$ , confirming answer C. Answer A) $2\sqrt{5}$ would result from incorrectly computing the norm. Answer B) $6\sqrt{3}$ might come from miscalculating $A^T A$ or the normal equation. Answer D) $0$ assumes the system has an exact solution, which it doesn't since we have 3 equations in 2 unknowns. Always remember: for least squares problems, set up the normal equation $A^T A \hat{x} = A^T b$ first, then compute the residual norm.
For the overdetermined system $A\mathbf{x} = \mathbf{b}$ where $A$ is $4 \times 2$ with rank 2, suppose the least squares solution is $\mathbf{x}^* = \begin{pmatrix} 3 \\ -1 \end{pmatrix}$ and the residual vector has length $\sqrt{6}$ . If we modify the system by replacing $\mathbf{b}$ with $\mathbf{b} + \mathbf{v}$ where $\mathbf{v}$ is in the column space of $A$ , what happens to the least squares solution?
Answer: The solution becomes $\mathbf{x}^* + (A^T A)^{-1}A^T \mathbf{v}$ and the residual length remains $\sqrt{6}$ When you encounter least squares problems with modifications to the right-hand side vector, focus on how changes affect the geometry of the solution space. The key insight is understanding what happens when you add a vector from the column space of $A$ to $\mathbf{b}$ . The new system becomes $A\mathbf{x} = \mathbf{b} + \mathbf{v}$ where $\mathbf{v} \in \text{Col}(A)$ . Since $\mathbf{v}$ is in the column space, there exists some vector $\mathbf{w}$ such that $\mathbf{v} = A\mathbf{w}$ . The new least squares solution satisfies $A^TA\mathbf{x}_{new} = A^T(\mathbf{b} + \mathbf{v})$ . Since $A^TA\mathbf{x}^* = A^T\mathbf{b}$ for the original solution, we get $\mathbf{x}_{new} = \mathbf{x}^* + (A^TA)^{-1}A^T\mathbf{v}$ . For the residual, the original residual $\mathbf{r} = \mathbf{b} - A\mathbf{x}^*$ is orthogonal to the column space. The new residual is $(\mathbf{b} + \mathbf{v}) - A\mathbf{x}_{new} = \mathbf{b} - A\mathbf{x}^* = \mathbf{r}$ , so its length remains $\sqrt{6}$ . Option A incorrectly adds $\|\mathbf{v}\|$ to the residual length, missing that $\mathbf{v}$ doesn't affect the orthogonal component. Option B keeps the wrong solution while also incorrectly changing the residual. Option D fails to account for how adding a vector in the column space shifts the solution. Remember: adding vectors in the column space of $A$ changes the least squares solution predictably via $(A^TA)^{-1}A^T$ , but the residual length stays constant because residuals are orthogonal to the column space.
Which equation represents the least-squares line $y = c_0 + c_1 x$ that best fits the data points $(-1, 0)$ , $(0, 1)$ , and $(1, 3)$ ?
Answer: $y = \frac{4}{3} + \frac{3}{2}x$ When you encounter a least-squares problem, you're finding the line that minimizes the sum of squared vertical distances between data points and the line. This requires setting up and solving a system of normal equations. To find the least-squares line $y = c_0 + c_1x$ for points $(-1, 0)$ , $(0, 1)$ , and $(1, 3)$ , you need to solve the system $A\mathbf{c} = \mathbf{b}$ where $A = \begin{pmatrix} 1 & -1 \\ 1 & 0 \\ 1 & 1 \end{pmatrix}$ and $\mathbf{b} = \begin{pmatrix} 0 \\ 1 \\ 3 \end{pmatrix}$ . The normal equations are $A^T A \mathbf{c} = A^T \mathbf{b}$ . Computing: $A^T A = \begin{pmatrix} 3 & 0 \\ 0 & 2 \end{pmatrix}$ and $A^T \mathbf{b} = \begin{pmatrix} 4 \\ 3 \end{pmatrix}$ . This gives us $3c_0 = 4$ and $2c_1 = 3$ , so $c_0 = \frac{4}{3}$ and $c_1 = \frac{3}{2}$ . Therefore, the least-squares line is $y = \frac{4}{3} + \frac{3}{2}x$ . Choice A gives $c_0 = 1$ and $c_1 = \frac{3}{2}$ , which has the wrong y-intercept. Choice B has $c_0 = \frac{3}{2}$ and $c_1 = \frac{4}{3}$ , swapping our correct coefficients. Choice C uses $c_1 = 2$ , which is the wrong slope. The correct answer is D. Study tip: For least-squares problems, always set up the matrix equation systematically and use the normal equations formula. Double-check by verifying that your line passes reasonably through the data points' general trend.
A data analyst fits a linear model $y = \alpha + \beta t$ to the points $(1,2)$ , $(2,3)$ , $(3,7)$ , and $(4,6)$ using least squares. If the resulting residuals are $r_1, r_2, r_3, r_4$ respectively, which relationship must hold?
Answer: $r_1 + r_2 + r_3 + r_4 = 0$ and $r_1 + 2r_2 + 3r_3 + 4r_4 = 0$ When you encounter least squares linear regression problems, focus on the fundamental properties that arise from the normal equations used to find the optimal parameters. In least squares regression with a model $y = \alpha + \beta t$ , the parameters $\alpha$ and $\beta$ are chosen to minimize $\sum_{i=1}^4 r_i^2$ , where $r_i = y_i - (\alpha + \beta t_i)$ are the residuals. This optimization leads to two crucial normal equations that the residuals must satisfy. The correct answer is D because least squares regression always produces residuals with two key properties. First, $\sum r_i = 0$ (the residuals sum to zero), and second, $\sum t_i r_i = 0$ (the residuals are orthogonal to the predictor variable). With our $t$ -values of 1, 2, 3, 4, this second condition becomes $r_1 + 2r_2 + 3r_3 + 4r_4 = 0$ . These are fundamental mathematical requirements of least squares, not just optimization goals. Option A is incorrect because least squares minimizes the sum of squared residuals, not the sum of absolute values or the maximum absolute value. Option B is only partially correct—while the sum of residuals equals zero and the sum of squared residuals is minimized, it's missing the crucial orthogonality condition. Option C is wrong because perfect fit (all residuals equal to zero) is extremely rare with real data and not a requirement of least squares. Remember: In least squares regression, always check for both orthogonality conditions—residuals must be orthogonal to the constant term (sum to zero) and to each predictor variable.

Start this quiz