Linear Regression via Least Squares - Linear Algebra

1. Find the least-squares solution $\hat{x}$ for the system $A\vec{x} = \vec{b}$ where $A = \begin{pmatrix} 1 & 0 \\ 1 & 1 \\ 1 & 2 \end{pmatrix}$ and $\vec{b} = \begin{pmatrix} 6 \\ 0 \\ 0 \end{pmatrix}$.
   Answer: $\begin{pmatrix} 5 \\ -3 \end{pmatrix}$ We must solve the normal equations $A^T A \hat{x} = A^T \vec{b}$. First, calculate the components: $A^T = \begin{pmatrix} 1 & 1 & 1 \\ 0 & 1 & 2 \end{pmatrix}$. $A^T A = \begin{pmatrix} 3 & 3 \\ 3 & 5 \end{pmatrix}$. $A^T \vec{b} = \begin{pmatrix} 1 & 1 & 1 \\ 0 & 1 & 2 \end{pmatrix} \begin{pmatrix} 6 \\ 0 \\ 0 \end{pmatrix} = \begin{pmatrix} 6 \\ 0 \end{pmatrix}$. The system to solve is $\begin{pmatrix} 3 & 3 \\ 3 & 5 \end{pmatrix} \begin{pmatrix} x_1 \\ x_2 \end{pmatrix} = \begin{pmatrix} 6 \\ 0 \end{pmatrix}$. From the first row, $3x_1 + 3x_2 = 6$, which simplifies to $x_1 + x_2 = 2$. From the second row, $3x_1 + 5x_2 = 0$. Substituting $x_1 = 2 - x_2$ into the second equation gives $3(2 - x_2) + 5x_2 = 0 \Rightarrow 6 - 3x_2 + 5x_2 = 0 \Rightarrow 2x_2 = -6 \Rightarrow x_2 = -3$. Then $x_1 = 2 - (-3) = 5$. The solution is $\hat{x} = \begin{pmatrix} 5 \\ -3 \end{pmatrix}$.
2. A researcher fits a line $y = \beta_0 + \beta_1 x$ to $n$ data points using least squares. The sum of squared residuals is $SSR = 45$, and the total sum of squares is $SST = 80$. If one additional data point is added that lies exactly on the fitted regression line, what happens to the coefficient of determination $R^2$?
   Answer: $R^2$ increases because the new point has zero residual, improving the overall fit quality When you encounter questions about how additional data points affect regression statistics, focus on how the coefficient of determination $R^2 = 1 - \frac{SSR}{SST}$ responds to changes in both the sum of squared residuals and total sum of squares. Initially, $R^2 = 1 - \frac{45}{80} = 0.4375$. When you add a point that lies exactly on the original fitted line, that new point has zero residual (perfect prediction), so $SSR$ remains 45. However, $SST$ increases because you're adding another data point's contribution to the total variability around the overall mean of all $n+1$ points. Since the denominator $SST$ increases while the numerator $SSR$ stays constant, the fraction $\frac{SSR}{SST}$ decreases, making $R^2$ larger. Option A is wrong because adding one point on the line doesn't create an underdetermined system—you still have more data points than parameters. Option B incorrectly assumes the regression line changes significantly and that any change increases variability, but a point on the original line actually demonstrates perfect fit for that observation. Option C misses the crucial insight that while $SSR$ doesn't change, $SST$ does increase, so the ratio changes even though the linear relationship appears unchanged. Study tip: Remember that $R^2$ depends on both $SSR$ and $SST$. When analyzing how additional data affects $R^2$, always consider how both components change, not just whether the new point fits well. Points with zero residuals always improve $R^2$ by increasing $SST$ without increasing $SSR$.
3. Two researchers analyze the same dataset but use different approaches: Researcher A centers the $x$-data by subtracting the mean before fitting $y = \beta_0 + \beta_1 x$, while Researcher B uses the original $x$-values. If the original data has $\bar{x} = 5$ and the least squares line using original data is $y = 2 + 0.8x$, what line will Researcher A obtain?
   Answer: $y = 6 + 0.8x_{centered}$ where $x_{centered} = x - 5$ When you encounter problems about data transformations in regression, the key insight is understanding how centering affects the intercept while leaving the slope unchanged. Let's work through what happens when Researcher A centers the data. The original regression line is $y = 2 + 0.8x$ with $\bar{x} = 5$. When we center the x-values by subtracting the mean, we create $x_{centered} = x - 5$, so $x = x_{centered} + 5$. Substituting this into the original equation: $y = 2 + 0.8(x_{centered} + 5) = 2 + 0.8x_{centered} + 4 = 6 + 0.8x_{centered}$ The slope remains 0.8 because centering is just a horizontal shift that doesn't change the relationship's steepness. The intercept becomes 6 because when $x_{centered} = 0$ (meaning $x = 5$, the original mean), we need $y = 2 + 0.8(5) = 6$. Answer A incorrectly keeps the original intercept of 2, missing that centering shifts where the line crosses the y-axis. Answer C gives -2 as the intercept, which would result from subtracting rather than adding the slope adjustment. Answer D not only has the wrong intercept but also incorrectly changes the slope to 0.64, suggesting a misunderstanding that centering affects the relationship's strength. Study tip: Remember that centering data only shifts the intercept—it never changes the slope. The new intercept equals the y-value when the centered variable equals zero (i.e., when the original variable equals its mean).
4. Let $\hat{x}$ be the least-squares solution to the system $A\vec{x} = \vec{b}$. The vector $A\hat{x}$ represents which of the following?
   Answer: The orthogonal projection of $\vec{b}$ onto the column space of $A$. The least-squares solution $\hat{x}$ is found such that $A\hat{x}$ is the vector in the column space of $A$ (denoted $\mathrm{Col}(A)$) that is closest to $\vec{b}$. This closest vector is the orthogonal projection of $\vec{b}$ onto $\mathrm{Col}(A)$. The error vector is $\vec{b} - A\hat{x}$, not $A\hat{x}$. The row space is not the correct subspace for the projection of $\vec{b}$. The solution vector $\hat{x}$ is in the domain $\mathbb{R}^n$, not the codomain where $\mathrm{Col}(A)$ resides.
5. To find the best-fit parabola of the form $y = a + bx^2$ (note the missing linear term) for a set of data points $(x_1, y_1), ..., (x_n, y_n)$, a linear system $A\vec{c} = \vec{y}$ is constructed. Which of the following correctly represents the design matrix $A$ for this model?
   Answer: $A = \begin{pmatrix} 1 & x_1^2 \\ 1 & x_2^2 \\ \vdots & \vdots \\ 1 & x_n^2 \end{pmatrix}$ The model equation is $y = a \cdot 1 + b \cdot x^2$. The coefficients to be found are $a$ and $b$. For each data point $(x_i, y_i)$, we get an equation $y_i = a \cdot 1 + b \cdot x_i^2$. The columns of the design matrix $A$ correspond to the functions of $x$ that are multiplied by the coefficients. In this case, they are $1$ and $x^2$. Therefore, the $i$-th row of $A$ should be $[1, x_i^2]$. Choice A is for a linear model $y=a+bx$. Choice C is for a full quadratic model $y=a+bx+cx^2$. Choice D incorrectly includes the dependent variable $y_i$ in the design matrix.
6. If the least-squares error for the system $A\vec{x} = \vec{b}$ is zero, what can be definitively concluded?
   Answer: The vector $\vec{b}$ is in the column space of $A$. The least-squares error is given by $\| \vec{b} - A\hat{x} \|$. If this error is zero, it means $\vec{b} - A\hat{x} = \vec{0}$, or $A\hat{x} = \vec{b}$. This implies that there exists a vector $\hat{x}$ such that $\vec{b}$ can be written as a linear combination of the columns of $A$. By definition, this means $\vec{b}$ is in the column space of $A$, and the system was consistent from the start. A non-trivial null space (A) could still exist if there are multiple solutions. The solution $\hat{x}$ (B) is only zero if $\vec{b}$ is zero (and columns of A are LI). The columns of $A$ (D) do not need to be orthonormal.
7. In a least squares regression with design matrix $A \in \mathbb{R}^{n \times p}$ where $n > p$ and $\text{rank}(A) = p$, the projection matrix is $P = A(A^T A)^{-1} A^T$. If a new data point $(\mathbf{x}_{new}, y_{new})$ is added where $\mathbf{x}_{new}$ is orthogonal to all columns of $A$, how does this affect the original least squares solution $\hat{\boldsymbol{\beta}}$?
   Answer: $\hat{\boldsymbol{\beta}}$ remains unchanged because $\mathbf{x}_{new}$ contributes zero information about the existing parameter relationships When you encounter questions about adding data points to least squares regression, focus on how the new information interacts with your existing parameter space. The key insight here is understanding what "orthogonal to all columns of $A$" means geometrically. The least squares solution $\hat{\boldsymbol{\beta}} = (A^T A)^{-1} A^T \mathbf{y}$ projects the response vector $\mathbf{y}$ onto the column space of $A$. When $\mathbf{x}_{new}$ is orthogonal to all columns of $A$, it lies entirely outside this column space. This means $\mathbf{x}_{new}$ cannot be expressed as any linear combination of the existing columns, so it provides no information about the relationships between $\mathbf{y}$ and the original features. Mathematically, since $\mathbf{x}_{new} \perp \text{col}(A)$, we have $A^T \mathbf{x}_{new} = \mathbf{0}$. The new point adds a residual that's orthogonal to the original fitted values, leaving the projection onto the original column space unchanged. Therefore, $\hat{\boldsymbol{\beta}}$ remains unchanged, making D correct. Option A incorrectly suggests a scaling effect from sample size, but least squares doesn't normalize by $n$. Option B misunderstands degrees of freedom—orthogonal directions don't affect existing parameter estimates. Option C is wrong because adding a linearly independent vector (which orthogonal $\mathbf{x}_{new}$ is) actually increases rank, not decreases it. Study tip: Remember that in regression, only the component of new data that lies within your existing feature space affects parameter estimates. Orthogonal components contribute only to unexplained variance, not to $\hat{\boldsymbol{\beta}}$.
8. Let $\hat{x}$ be the least-squares solution to an inconsistent system $A\vec{x} = \vec{b}$. If $\vec{e} = \vec{b} - A\hat{x}$ is the corresponding error vector, which of the following statements about $\vec{e}$ is always true?
   Answer: $\vec{e}$ is orthogonal to the column space of $A$. The fundamental principle of least-squares is that the error vector $\vec{e} = \vec{b} - A\hat{x}$ must be orthogonal to the subspace onto which we are projecting, which is the column space of $A$. This condition, $A^T(\vec{b} - A\hat{x}) = \vec{0}$, gives rise to the normal equations. The error vector is only the zero vector if the system is consistent, which it is not. There is no guarantee that $\vec{e}$ is orthogonal to $\vec{b}$. Since $\vec{e}$ is orthogonal to $\mathrm{Col}(A)$, it cannot be an element of it (unless it is the zero vector).
9. A system $A\vec{x} = \vec{b}$ is inconsistent. Which of the following best describes the goal of solving the associated normal equations, $A^T A \hat{x} = A^T \vec{b}$?
   Answer: To find a vector $\hat{x}$ that is an exact solution to a modified system where $\vec{b}$ is replaced by its projection onto $\mathrm{Col}(A)$. When you encounter an inconsistent system $A\vec{x} = \vec{b}$, you're dealing with a fundamental problem: the vector $\vec{b}$ doesn't lie in the column space of $A$, so no exact solution exists. The normal equations $A^T A \hat{x} = A^T \vec{b}$ provide the best possible alternative by finding the least squares solution. The key insight is understanding what "best possible" means geometrically. Since $\vec{b}$ isn't in $\text{Col}(A)$, we find the closest vector that is in $\text{Col}(A)$ – this is the orthogonal projection of $\vec{b}$ onto $\text{Col}(A)$, denoted $\text{proj}_{\text{Col}(A)}\vec{b}$. The normal equations actually solve the modified system $A\hat{x} = \text{proj}_{\text{Col}(A)}\vec{b}$, which has an exact solution because the projection lies in the column space. Option D correctly captures this: we're solving a system where $\vec{b}$ is replaced by its projection onto $\text{Col}(A)$. Option A is wrong because minimizing $||\hat{x}||$ isn't the goal – we minimize $||A\vec{x} - \vec{b}||$. Option B misunderstands the objective; we're not trying to satisfy individual equations but rather minimizing the overall residual. Option C is incorrect because the normal equations don't involve orthogonalizing the columns of $A$ – that would be a different approach like QR decomposition. Remember: the normal equations always transform an inconsistent system into a consistent one by projecting $\vec{b}$ onto the column space, making exact solution possible.
10. Consider finding the least-squares solution $\hat{x}$ to $A\vec{x} = \vec{b}$. Let $\hat{b} = A\hat{x}$ be the best approximation of $\vec{b}$ in the column space of $A$. Which of the following statements is always true?
    Answer: The vector $\vec{b} - \hat{b}$ is orthogonal to $\hat{b}$. When you encounter least-squares problems, you're working with orthogonal projections. The key insight is understanding the geometric relationship between the original vector $\vec{b}$, its projection $\hat{b}$ onto the column space of $A$, and the error vector $\vec{b} - \hat{b}$. The fundamental property of orthogonal projection is that the error vector (the difference between the original vector and its projection) is always orthogonal to the projection itself. This is because $\hat{b} = A\hat{x}$ lies in the column space of $A$, and the least-squares solution minimizes $||\vec{b} - A\vec{x}||^2$ by ensuring the residual $\vec{b} - \hat{b}$ is orthogonal to the entire column space. Since $\hat{b}$ is in this column space, $\vec{b} - \hat{b}$ must be orthogonal to $\hat{b}$. This confirms answer D is correct. Let's examine why the other options fail. Option A claims $\hat{b}$ is orthogonal to $\vec{b}$, but this only occurs in the special case where $\vec{b}$ is entirely outside the column space. Generally, $\hat{b}$ and $\vec{b}$ have a non-zero dot product. Option B suggests equal norms, but $||\hat{b}|| \leq ||\vec{b}||$ with equality only when $\vec{b}$ is already in the column space. Option C incorrectly relates $\hat{x}$ to $\hat{b}$—there's no required orthogonality between the solution vector and the projection. Remember: in least-squares problems, always think about the orthogonal decomposition $\vec{b} = \hat{b} + (\vec{b} - \hat{b})$, where the projection and error components are perpendicular.
11. A researcher collects data points $(1, 2), (2, 5), (3, 7), (4, 11)$ and wants to fit a line $y = ax + b$ using least squares. If the normal equations yield the system $\begin{bmatrix} 30 & 10 \\ 10 & 4 \end{bmatrix} \begin{bmatrix} a \\ b \end{bmatrix} = \begin{bmatrix} 91 \\ 25 \end{bmatrix}$, what is the slope $a$ of the least squares line?
    Answer: $a = 2.9$ Solving the normal equations: From the second equation, $10a + 4b = 25$, so $b = \frac{25 - 10a}{4}$. Substituting into the first equation: $30a + 10 \cdot \frac{25 - 10a}{4} = 91$. This gives $30a + \frac{250 - 100a}{4} = 91$, which simplifies to $30a + 62.5 - 25a = 91$, so $5a = 28.5$ and $a = 2.9$. Choice B results from arithmetic errors in fraction manipulation. Choice C comes from incorrectly using $b = \frac{25 - 10a}{10}$ instead of dividing by 4. Choice D results from sign errors in the substitution.
12. In a weighted least squares problem, the objective is to minimize $\sum_{i=1}^n w_i (y_i - \beta_0 - \beta_1 x_i)^2$ where $w_i > 0$. If the weights are $w_1 = 4, w_2 = 1, w_3 = 9$ for data points $(1, 2), (2, 5), (3, 7)$, what is the effective design matrix $\tilde{A}$ in the transformed problem $\min \|\tilde{A}\boldsymbol{\beta} - \tilde{\mathbf{b}}\|^2$?
    Answer: $\tilde{A} = \begin{bmatrix} 2 & 2 \\ 1 & 2 \\ 3 & 9 \end{bmatrix}$ When you encounter weighted least squares problems, the key insight is transforming the weighted problem into a standard least squares form by incorporating the square roots of the weights into both the design matrix and response vector. Starting with the original design matrix for the linear model $y = \beta_0 + \beta_1 x$, you have: $A = \begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix}$ To transform this into the standard form $\min \|\tilde{A}\boldsymbol{\beta} - \tilde{\mathbf{b}}\|^2$, you multiply each row by $\sqrt{w_i}$. With weights $w_1 = 4, w_2 = 1, w_3 = 9$, the square roots are $\sqrt{4} = 2, \sqrt{1} = 1, \sqrt{9} = 3$. This gives you: $\tilde{A} = \begin{bmatrix} 2 \cdot 1 & 2 \cdot 1 \\ 1 \cdot 1 & 1 \cdot 2 \\ 3 \cdot 1 & 3 \cdot 3 \end{bmatrix} = \begin{bmatrix} 2 & 2 \\ 1 & 2 \\ 3 & 9 \end{bmatrix}$ This matches answer choice B. Choice A incorrectly uses $\sqrt{w_3} = 9$ instead of $3$ in the last row. Choice C uses the original weights instead of their square roots ($w_1 = 4$ instead of $\sqrt{w_1} = 2$). Choice D appears to use $2\sqrt{w_1} = 8$ in the first row, suggesting confusion about how to apply the transformation. Remember: in weighted least squares transformations, always use the square root of the weights to scale each row of both the design matrix and response vector. This converts the weighted problem into an equivalent unweighted problem that preserves the underlying linear algebra structure.
13. What is the slope of the least-squares regression line for the data points $(-1, 0)$, $(0, 1)$, $(1, 3)$, and $(2, 4)$?
    Answer: $1.4$ The model is $y = c_0 + c_1 x$. The matrix system is $A\vec{c} = \vec{b}$ with $A = \begin{pmatrix} 1 & -1 \\ 1 & 0 \\ 1 & 1 \\ 1 & 2 \end{pmatrix}$ and $\vec{b} = \begin{pmatrix} 0 \\ 1 \\ 3 \\ 4 \end{pmatrix}$. The normal equations are $A^T A \vec{c} = A^T \vec{b}$. We compute $A^T A = \begin{pmatrix} 4 & 2 \\ 2 & 6 \end{pmatrix}$ and $A^T \vec{b} = \begin{pmatrix} 8 \\ 11 \end{pmatrix}$. We must solve $\begin{pmatrix} 4 & 2 \\ 2 & 6 \end{pmatrix} \begin{pmatrix} c_0 \\ c_1 \end{pmatrix} = \begin{pmatrix} 8 \\ 11 \end{pmatrix}$. The second row gives $2c_0 + 6c_1 = 11$. The first row gives $4c_0 + 2c_1 = 8$, or $2c_0 + c_1 = 4$. Subtracting the two equations: $(2c_0 + 6c_1) - (2c_0 + c_1) = 11 - 4$, which simplifies to $5c_1 = 7$, so $c_1 = 1.4$. The slope is $c_1 = 1.4$.
14. Consider two least squares problems: Problem 1 fits $y = a + bx$ to points $(1, 2), (2, 4), (3, 6)$, and Problem 2 fits $y = cx + d$ to the same points. The normal equations for Problem 1 yield $\begin{bmatrix} 3 & 6 \\ 6 & 14 \end{bmatrix} \begin{bmatrix} a \\ b \end{bmatrix} = \begin{bmatrix} 12 \\ 28 \end{bmatrix}$. What is the key difference in the solutions?
    Answer: Both problems yield identical solutions since they represent the same linear model with reordered parameters When you encounter least squares problems with different parameter orderings, focus on whether the underlying mathematical model actually changes. Both problems are fitting the exact same linear relationship $y = mx + c$ to the same data points—they just use different variable names and orderings. Problem 1 uses $y = a + bx$ (intercept first, slope second) while Problem 2 uses $y = cx + d$ (slope first, intercept second). Since the data points $(1,2), (2,4), (3,6)$ lie perfectly on the line $y = 2x$, both problems will find the same line: intercept = 0, slope = 2. The only difference is that Problem 1 gives you $a = 0, b = 2$ while Problem 2 gives you $c = 2, d = 0$. Choice A is incorrect because both problems have unique solutions—reordering parameters doesn't change the mathematical properties of the system. Choice B misunderstands parameter ordering: the slope value itself doesn't change, only which variable represents it. The numerical conditioning of the normal equations remains the same since you're solving the same geometric problem. Choice D is wrong because identical models fitting identical data must produce identical residuals—the residual sum of squares measures the quality of fit, which is unchanged by parameter relabeling. Choice C correctly recognizes that these represent the same linear model with reordered parameters, yielding identical solutions. Study tip: When comparing least squares formulations, always check if they represent the same underlying model before analyzing differences in conditioning or solution properties.
15. A researcher wants to find the best-fit line of the form $y = c_0 + c_1 x$ for the data points $(0, 1)$, $(1, 3)$, and $(2, 4)$. The least-squares solution $\hat{c} = \begin{pmatrix} c_0 \\ c_1 \end{pmatrix}$ is found by solving the normal equations $A^T A \hat{c} = A^T \vec{b}$. What is the matrix $A^T A$ for this problem?
    Answer: $\begin{pmatrix} 3 & 3 \\ 3 & 5 \end{pmatrix}$ First, construct the design matrix $A$ and the observation vector $\vec{b}$. The $i$-th row of $A$ is $[1, x_i]$. So, $A = \begin{pmatrix} 1 & 0 \\ 1 & 1 \\ 1 & 2 \end{pmatrix}$ and $\vec{b} = \begin{pmatrix} 1 \\ 3 \\ 4 \end{pmatrix}$. Next, find the transpose $A^T = \begin{pmatrix} 1 & 1 & 1 \\ 0 & 1 & 2 \end{pmatrix}$. Finally, compute the product $A^T A$: $A^T A = \begin{pmatrix} 1 & 1 & 1 \\ 0 & 1 & 2 \end{pmatrix} \begin{pmatrix} 1 & 0 \\ 1 & 1 \\ 1 & 2 \end{pmatrix} = \begin{pmatrix} (1+1+1) & (0+1+2) \\ (0+1+2) & (0+1+4) \end{pmatrix} = \begin{pmatrix} 3 & 3 \\ 3 & 5 \end{pmatrix}$. Choice A is $AA^T$. Choice C results from incorrectly setting up $A$ with columns swapped. Choice D is the matrix $A$ itself.
16. A dataset has design matrix $A$ and response vector $\mathbf{b}$. If $\mathbf{b}$ lies in the column space of $A$, and the least squares solution is $\hat{\mathbf{x}} = (A^T A)^{-1} A^T \mathbf{b}$, what is the relationship between the residual vector $\mathbf{r} = \mathbf{b} - A\hat{\mathbf{x}}$ and the fitted values $A\hat{\mathbf{x}}$?
    Answer: $\mathbf{r} = \mathbf{0}$ and $A\hat{\mathbf{x}} = \mathbf{b}$, since $\mathbf{b}$ is exactly representable in the column space If $\mathbf{b}$ lies in the column space of $A$, then there exists an exact solution $\mathbf{x}_0$ such that $A\mathbf{x}_0 = \mathbf{b}$. The least squares solution $\hat{\mathbf{x}} = (A^T A)^{-1} A^T \mathbf{b}$ will find this exact solution, making $A\hat{\mathbf{x}} = \mathbf{b}$ and thus $\mathbf{r} = \mathbf{b} - A\hat{\mathbf{x}} = \mathbf{0}$. Choice B describes the general orthogonality property but doesn't account for the special case where $\mathbf{b}$ is in the column space. Choice C is incorrect because $\mathbf{r}$ lies in the null space of $A^T$, not the column space of $A$. Choice D gives an arbitrary relationship that doesn't follow from projection theory.
17. The least-squares solution $\hat{x}$ to the system $A\vec{x} = \vec{b}$ is unique if and only if which condition is met?
    Answer: The columns of $A$ are linearly independent. The least-squares solution is found by solving the normal equations $A^T A \hat{x} = A^T \vec{b}$. This system has a unique solution if and only if the matrix $A^T A$ is invertible. The matrix $A^T A$ is invertible if and only if the columns of $A$ are linearly independent. If $A$ is square and invertible (B), there is a unique exact solution, which is a special case. Orthogonal columns (C) is a sufficient condition for linear independence, but not a necessary one. If $\vec{b}$ is in $\mathrm{Col}(A)$ (D), an exact solution exists, but it may not be unique if the columns of $A$ are linearly dependent.
18. Let $\hat{x}$ be a least-squares solution to an inconsistent system $A\vec{x} = \vec{b}$. Which of the following conditions is NOT necessarily satisfied by $\hat{x}$?
    Answer: $A(\vec{b} - A\hat{x}) = \vec{0}$ The defining property of the least-squares solution is that the error vector, $\vec{e} = \vec{b} - A\hat{x}$, is orthogonal to the column space of $A$. This orthogonality condition is expressed as $A^T\vec{e} = \vec{0}$, which is choice A. Choice D is the definition of minimizing the residual norm. Choice C is the geometric interpretation of the result. Choice B, $A(\vec{b} - A\hat{x}) = \vec{0}$, states that the error vector is in the null space of $A$. This is not generally true; the error vector is in the null space of $A^T$ (the orthogonal complement of the column space of $A$).
19. A least squares regression produces the fitted model $\hat{y} = 2.5 + 1.8x$ with residuals $r_1 = 0.3, r_2 = -0.7, r_3 = 0.4$. If the original $y$-values are transformed by $y_{new} = 2y - 3$, what will be the residuals for the regression of $y_{new}$ on $x$?
    Answer: $r_{1,new} = 0.6, r_{2,new} = -1.4, r_{3,new} = 0.8$ When you encounter questions about transforming data in regression, the key insight is understanding how linear transformations affect both fitted values and residuals in predictable ways. A residual is the difference between an observed value and its fitted value: $r_i = y_i - \hat{y}_i$. When you transform the response variable by $y_{new} = 2y - 3$, both the observed values and fitted values get transformed by the same rule. This means $y_{i,new} = 2y_i - 3$ and $\hat{y}_{i,new} = 2\hat{y}_i - 3$. The new residuals become: $r_{i,new} = y_{i,new} - \hat{y}_{i,new} = (2y_i - 3) - (2\hat{y}_i - 3) = 2y_i - 2\hat{y}_i = 2(y_i - \hat{y}_i) = 2r_i$. The constant term (-3) cancels out completely, and only the multiplicative factor (2) affects the residuals. Therefore: $r_{1,new} = 2(0.3) = 0.6$, $r_{2,new} = 2(-0.7) = -1.4$, and $r_{3,new} = 2(0.4) = 0.8$. Answer C is correct. Answer A incorrectly assumes residuals remain unchanged after transformation. Answer B appears to subtract 3 from each doubled residual, wrongly applying the constant term. Answer D seems to add values nonsensically without following any transformation rule. Remember: when you transform $y$ by $ay + b$, residuals get multiplied by $a$ only—the constant $b$ always cancels out because it affects both observed and fitted values equally.
20. Consider the least squares problem for fitting $y = c_1 x + c_2 x^2$ to data points $(1, 3), (2, 8), (3, 15)$. The design matrix $A$ has the property that $A^T A$ is singular. What can be concluded about this least squares problem?
    Answer: The normal equations have infinitely many solutions, indicating the model is overparameterized for this dataset When $A^T A$ is singular, the normal equations $A^T A \mathbf{c} = A^T \mathbf{b}$ have either no solution or infinitely many solutions. Since we're fitting 2 parameters to 3 data points with a reasonable model, the system is typically consistent, leading to infinitely many solutions. This indicates linear dependence in the columns of $A$, meaning the model is overparameterized. Choice B is wrong because inconsistency relates to $A^T \mathbf{b}$ not being in the column space of $A^T A$, which is unlikely here. Choice C confuses singularity with perfect fitting. Choice D overstates the problem - while regularization could help, the standard approach doesn't 'fail completely.'

Start this quiz