Linear Algebra Quiz: Linear Regression Via Least Squares

20 questions · exam conditions

0:00

Linear Regression Via Least SquaresQuestion 1 of 20

A researcher wants to find the best-fit line of the form $y = c_0 + c_1 x$ for the data points $(0, 1)$ , $(1, 3)$ , and $(2, 4)$ . The least-squares solution $\hat{c} = \begin{pmatrix} c_0 \\ c_1 \end{pmatrix}$ is found by solving the normal equations $A^T A \hat{c} = A^T \vec{b}$ . What is the matrix $A^T A$ for this problem?

\begin{pmatrix} 1 & 1 & 1 \\ 1 & 2 & 3 \\ 1 & 3 & 5 \end{pmatrix}

\begin{pmatrix} 3 & 3 \\ 3 & 5 \end{pmatrix}

\begin{pmatrix} 5 & 3 \\ 3 & 3 \end{pmatrix}

\begin{pmatrix} 1 & 0 \\ 1 & 1 \\ 1 & 2 \end{pmatrix}

← Back to quizzes

Linear Algebra Quiz

Linear Algebra Quiz: Linear Regression Via Least Squares

Practice Linear Regression Via Least Squares in Linear Algebra with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on Linear Regression Via Least Squares, giving you a quick way to practice the rules, question types, and explanations that matter most for Linear Algebra.

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

$\begin{pmatrix} 1 & 1 & 1 \\ 1 & 2 & 3 \\ 1 & 3 & 5 \end{pmatrix}$
$\begin{pmatrix} 3 & 3 \\ 3 & 5 \end{pmatrix}$ (correct answer)
$\begin{pmatrix} 5 & 3 \\ 3 & 3 \end{pmatrix}$
$\begin{pmatrix} 1 & 0 \\ 1 & 1 \\ 1 & 2 \end{pmatrix}$

Explanation: First, construct the design matrix

A

and the observation vector

\vec{b}

. The

i

-th row of

A

[1, x_i]

. So,

A = \begin{pmatrix} 1 & 0 \\ 1 & 1 \\ 1 & 2 \end{pmatrix}

and

\vec{b} = \begin{pmatrix} 1 \\ 3 \\ 4 \end{pmatrix}

. Next, find the transpose

A^T = \begin{pmatrix} 1 & 1 & 1 \\ 0 & 1 & 2 \end{pmatrix}

. Finally, compute the product

A^T A

A^T A = \begin{pmatrix} 1 & 1 & 1 \\ 0 & 1 & 2 \end{pmatrix} \begin{pmatrix} 1 & 0 \\ 1 & 1 \\ 1 & 2 \end{pmatrix} = \begin{pmatrix} (1+1+1) & (0+1+2) \\ (0+1+2) & (0+1+4) \end{pmatrix} = \begin{pmatrix} 3 & 3 \\ 3 & 5 \end{pmatrix}

. Choice A is

AA^T

. Choice C results from incorrectly setting up

A

with columns swapped. Choice D is the matrix

A

itself.

Question 2

Consider finding the least-squares solution $\hat{x}$ to $A\vec{x} = \vec{b}$ . Let $\hat{b} = A\hat{x}$ be the best approximation of $\vec{b}$ in the column space of $A$ . Which of the following statements is always true?

$\hat{b}$ is orthogonal to $\vec{b}$ .
The norm of $\hat{b}$ is equal to the norm of $\vec{b}$ .
The solution vector $\hat{x}$ is orthogonal to $\hat{b}$ .
The vector $\vec{b} - \hat{b}$ is orthogonal to $\hat{b}$ . (correct answer)

Explanation: When you encounter least-squares problems, you're working with orthogonal projections. The key insight is understanding the geometric relationship between the original vector

\vec{b}

, its projection

\hat{b}

onto the column space of

A

, and the error vector

\vec{b} - \hat{b}

. The fundamental property of orthogonal projection is that the error vector (the difference between the original vector and its projection) is always orthogonal to the projection itself. This is because

\hat{b} = A\hat{x}

lies in the column space of

A

, and the least-squares solution minimizes

||\vec{b} - A\vec{x}||^2

by ensuring the residual

\vec{b} - \hat{b}

is orthogonal to the entire column space. Since

\hat{b}

is in this column space,

\vec{b} - \hat{b}

must be orthogonal to

\hat{b}

. This confirms answer D is correct. Let's examine why the other options fail. Option A claims

\hat{b}

is orthogonal to

\vec{b}

, but this only occurs in the special case where

\vec{b}

is entirely outside the column space. Generally,

\hat{b}

and

\vec{b}

have a non-zero dot product. Option B suggests equal norms, but

||\hat{b}|| \leq ||\vec{b}||

with equality only when

\vec{b}

is already in the column space. Option C incorrectly relates

\hat{x}

\hat{b}

—there's no required orthogonality between the solution vector and the projection. Remember: in least-squares problems, always think about the orthogonal decomposition

\vec{b} = \hat{b} + (\vec{b} - \hat{b})

, where the projection and error components are perpendicular.

Question 3

A system $A\vec{x} = \vec{b}$ is inconsistent. Which of the following best describes the goal of solving the associated normal equations, $A^T A \hat{x} = A^T \vec{b}$ ?

To find a vector $\hat{x}$ whose norm is as small as possible.
To find a vector $\hat{x}$ that satisfies as many of the original equations in $A\vec{x} = \vec{b}$ as possible.
To find a vector $\hat{x}$ that is an exact solution to a modified system where the columns of $A$ are orthogonalized.
To find a vector $\hat{x}$ that is an exact solution to a modified system where $\vec{b}$ is replaced by its projection onto $\mathrm{Col}(A)$ . (correct answer)

Explanation: When you encounter an inconsistent system

A\vec{x} = \vec{b}

, you're dealing with a fundamental problem: the vector

\vec{b}

doesn't lie in the column space of

A

, so no exact solution exists. The normal equations

A^T A \hat{x} = A^T \vec{b}

provide the best possible alternative by finding the least squares solution. The key insight is understanding what "best possible" means geometrically. Since

\vec{b}

isn't in

\text{Col}(A)

, we find the closest vector that is in

\text{Col}(A)

– this is the orthogonal projection of

\vec{b}

onto

\text{Col}(A)

, denoted

\text{proj}_{\text{Col}(A)}\vec{b}

. The normal equations actually solve the modified system

A\hat{x} = \text{proj}_{\text{Col}(A)}\vec{b}

, which has an exact solution because the projection lies in the column space. Option D correctly captures this: we're solving a system where

\vec{b}

is replaced by its projection onto

\text{Col}(A)

. Option A is wrong because minimizing

||\hat{x}||

isn't the goal – we minimize

||A\vec{x} - \vec{b}||

. Option B misunderstands the objective; we're not trying to satisfy individual equations but rather minimizing the overall residual. Option C is incorrect because the normal equations don't involve orthogonalizing the columns of

A

– that would be a different approach like QR decomposition. Remember: the normal equations always transform an inconsistent system into a consistent one by projecting

\vec{b}

onto the column space, making exact solution possible.

Question 4

Let $\hat{x}$ be the least-squares solution to an inconsistent system $A\vec{x} = \vec{b}$ . If $\vec{e} = \vec{b} - A\hat{x}$ is the corresponding error vector, which of the following statements about $\vec{e}$ is always true?

$\vec{e}$ is orthogonal to $\vec{b}$ .
$\vec{e}$ is an element of the column space of $A$ .
$\vec{e}$ is orthogonal to the column space of $A$ . (correct answer)
$\vec{e}$ is the zero vector.

Explanation: The fundamental principle of least-squares is that the error vector

\vec{e} = \vec{b} - A\hat{x}

must be orthogonal to the subspace onto which we are projecting, which is the column space of

A

. This condition,

A^T(\vec{b} - A\hat{x}) = \vec{0}

, gives rise to the normal equations. The error vector is only the zero vector if the system is consistent, which it is not. There is no guarantee that

\vec{e}

is orthogonal to

\vec{b}

. Since

\vec{e}

is orthogonal to

\mathrm{Col}(A)

, it cannot be an element of it (unless it is the zero vector).

Question 5

To find the best-fit parabola of the form $y = a + bx^2$ (note the missing linear term) for a set of data points $(x_1, y_1), ..., (x_n, y_n)$ , a linear system $A\vec{c} = \vec{y}$ is constructed. Which of the following correctly represents the design matrix $A$ for this model?

(correct answer)

Explanation: The model equation is

y = a \cdot 1 + b \cdot x^2

. The coefficients to be found are

a

and

b

. For each data point

(x_i, y_i)

, we get an equation

y_i = a \cdot 1 + b \cdot x_i^2

. The columns of the design matrix

A

correspond to the functions of

x

that are multiplied by the coefficients. In this case, they are

1

and

x^2

. Therefore, the

i

-th row of

A

should be

[1, x_i^2]

. Choice A is for a linear model

y=a+bx

. Choice C is for a full quadratic model

y=a+bx+cx^2

. Choice D incorrectly includes the dependent variable

y_i

in the design matrix.

Question 6

Find the least-squares solution $\hat{x}$ for the system $A\vec{x} = \vec{b}$ where $A = \begin{pmatrix} 1 & 0 \\ 1 & 1 \\ 1 & 2 \end{pmatrix}$ and $\vec{b} = \begin{pmatrix} 6 \\ 0 \\ 0 \end{pmatrix}$ .

$\begin{pmatrix} 6 \\ -6 \end{pmatrix}$
$\begin{pmatrix} -1 \\ 3 \end{pmatrix}$
$\begin{pmatrix} 30 \\ -18 \end{pmatrix}$
$\begin{pmatrix} 5 \\ -3 \end{pmatrix}$ (correct answer)

Explanation: We must solve the normal equations

A^T A \hat{x} = A^T \vec{b}

. First, calculate the components:

A^T = \begin{pmatrix} 1 & 1 & 1 \\ 0 & 1 & 2 \end{pmatrix}

A^T A = \begin{pmatrix} 3 & 3 \\ 3 & 5 \end{pmatrix}

A^T \vec{b} = \begin{pmatrix} 1 & 1 & 1 \\ 0 & 1 & 2 \end{pmatrix} \begin{pmatrix} 6 \\ 0 \\ 0 \end{pmatrix} = \begin{pmatrix} 6 \\ 0 \end{pmatrix}

. The system to solve is

\begin{pmatrix} 3 & 3 \\ 3 & 5 \end{pmatrix} \begin{pmatrix} x_1 \\ x_2 \end{pmatrix} = \begin{pmatrix} 6 \\ 0 \end{pmatrix}

. From the first row,

3x_1 + 3x_2 = 6

, which simplifies to

x_1 + x_2 = 2

. From the second row,

3x_1 + 5x_2 = 0

. Substituting

x_1 = 2 - x_2

into the second equation gives

3(2 - x_2) + 5x_2 = 0 \Rightarrow 6 - 3x_2 + 5x_2 = 0 \Rightarrow 2x_2 = -6 \Rightarrow x_2 = -3

. Then

x_1 = 2 - (-3) = 5

. The solution is

\hat{x} = \begin{pmatrix} 5 \\ -3 \end{pmatrix}

Question 7

Consider the least squares problem for fitting $y = c_1 x + c_2 x^2$ to data points $(1, 3), (2, 8), (3, 15)$ . The design matrix $A$ has the property that $A^T A$ is singular. What can be concluded about this least squares problem?

The normal equations have infinitely many solutions, indicating the model is overparameterized for this dataset (correct answer)
The least squares solution does not exist because the system is inconsistent with the given constraints
The residual sum of squares will be exactly zero, indicating a perfect fit to the data points
The problem requires regularization techniques since the standard least squares approach will fail completely

Explanation: When

A^T A

is singular, the normal equations

A^T A \mathbf{c} = A^T \mathbf{b}

have either no solution or infinitely many solutions. Since we're fitting 2 parameters to 3 data points with a reasonable model, the system is typically consistent, leading to infinitely many solutions. This indicates linear dependence in the columns of

A

, meaning the model is overparameterized. Choice B is wrong because inconsistency relates to

A^T \mathbf{b}

not being in the column space of

A^T A

, which is unlikely here. Choice C confuses singularity with perfect fitting. Choice D overstates the problem - while regularization could help, the standard approach doesn't 'fail completely.'

Question 8

A researcher fits a line $y = \beta_0 + \beta_1 x$ to $n$ data points using least squares. The sum of squared residuals is $SSR = 45$ , and the total sum of squares is $SST = 80$ . If one additional data point is added that lies exactly on the fitted regression line, what happens to the coefficient of determination $R^2$ ?

$R^2$ becomes undefined because the regression parameters are no longer uniquely determined
$R^2$ decreases because adding any point changes the regression line and increases variability
$R^2$ remains the same because the point lies on the original line and doesn't change the relationship
$R^2$ increases because the new point has zero residual, improving the overall fit quality (correct answer)

Explanation: When you encounter questions about how additional data points affect regression statistics, focus on how the coefficient of determination

R^2 = 1 - \frac{SSR}{SST}

responds to changes in both the sum of squared residuals and total sum of squares. Initially,

R^2 = 1 - \frac{45}{80} = 0.4375

. When you add a point that lies exactly on the original fitted line, that new point has zero residual (perfect prediction), so

SSR

remains 45. However,

SST

increases because you're adding another data point's contribution to the total variability around the overall mean of all

n+1

points. Since the denominator

SST

increases while the numerator

SSR

stays constant, the fraction

\frac{SSR}{SST}

decreases, making

R^2

larger. Option A is wrong because adding one point on the line doesn't create an underdetermined system—you still have more data points than parameters. Option B incorrectly assumes the regression line changes significantly and that any change increases variability, but a point on the original line actually demonstrates perfect fit for that observation. Option C misses the crucial insight that while

SSR

doesn't change,

SST

does increase, so the ratio changes even though the linear relationship appears unchanged. Study tip: Remember that

R^2

depends on both

SSR

and

SST

. When analyzing how additional data affects

R^2

, always consider how both components change, not just whether the new point fits well. Points with zero residuals always improve

R^2

by increasing

SST

without increasing

SSR

Question 9

Consider two least squares problems: Problem 1 fits $y = a + bx$ to points $(1, 2), (2, 4), (3, 6)$ , and Problem 2 fits $y = cx + d$ to the same points. The normal equations for Problem 1 yield $$ \begin{bmatrix} 3 & 6 \ 6 & 14 \end{bmatrix} \begin{bmatrix} a \ b \end{bmatrix} = \begin{bmatrix} 12 \ 28 \end{bmatrix}

Problem 1 has a unique solution while Problem 2 may have infinitely many solutions due to different parameterization
Problem 2 will have a different slope because the parameter ordering affects the numerical conditioning
Both problems yield identical solutions since they represent the same linear model with reordered parameters (correct answer)
The residual sum of squares will be different between the two problems despite fitting the same data

Explanation: When you encounter least squares problems with different parameter orderings, focus on whether the underlying mathematical model actually changes. Both problems are fitting the exact same linear relationship

y = mx + c

to the same data points—they just use different variable names and orderings. Problem 1 uses

y = a + bx

(intercept first, slope second) while Problem 2 uses

y = cx + d

(slope first, intercept second). Since the data points

(1,2), (2,4), (3,6)

lie perfectly on the line

y = 2x

, both problems will find the same line: intercept = 0, slope = 2. The only difference is that Problem 1 gives you

a = 0, b = 2

while Problem 2 gives you

c = 2, d = 0

. Choice A is incorrect because both problems have unique solutions—reordering parameters doesn't change the mathematical properties of the system. Choice B misunderstands parameter ordering: the slope value itself doesn't change, only which variable represents it. The numerical conditioning of the normal equations remains the same since you're solving the same geometric problem. Choice D is wrong because identical models fitting identical data must produce identical residuals—the residual sum of squares measures the quality of fit, which is unchanged by parameter relabeling. Choice C correctly recognizes that these represent the same linear model with reordered parameters, yielding identical solutions. Study tip: When comparing least squares formulations, always check if they represent the same underlying model before analyzing differences in conditioning or solution properties.

Question 10

A least squares regression produces the fitted model $\hat{y} = 2.5 + 1.8x$ with residuals $r_1 = 0.3, r_2 = -0.7, r_3 = 0.4$ . If the original $y$ -values are transformed by $y_{new} = 2y - 3$ , what will be the residuals for the regression of $y_{new}$ on $x$ ?

$r_{1,new} = 0.3, r_{2,new} = -0.7, r_{3,new} = 0.4$
$r_{1,new} = -2.4, r_{2,new} = -4.4, r_{3,new} = -2.2$
$r_{1,new} = 0.6, r_{2,new} = -1.4, r_{3,new} = 0.8$ (correct answer)
$r_{1,new} = 2.8, r_{2,new} = 1.8, r_{3,new} = 2.9$

Explanation: When you encounter questions about transforming data in regression, the key insight is understanding how linear transformations affect both fitted values and residuals in predictable ways. A residual is the difference between an observed value and its fitted value:

r_i = y_i - \hat{y}_i

. When you transform the response variable by

y_{new} = 2y - 3

, both the observed values and fitted values get transformed by the same rule. This means

y_{i,new} = 2y_i - 3

and

\hat{y}_{i,new} = 2\hat{y}_i - 3

. The new residuals become:

r_{i,new} = y_{i,new} - \hat{y}_{i,new} = (2y_i - 3) - (2\hat{y}_i - 3) = 2y_i - 2\hat{y}_i = 2(y_i - \hat{y}_i) = 2r_i

. The constant term (-3) cancels out completely, and only the multiplicative factor (2) affects the residuals. Therefore:

r_{1,new} = 2(0.3) = 0.6

r_{2,new} = 2(-0.7) = -1.4

, and

r_{3,new} = 2(0.4) = 0.8

. Answer C is correct. Answer A incorrectly assumes residuals remain unchanged after transformation. Answer B appears to subtract 3 from each doubled residual, wrongly applying the constant term. Answer D seems to add values nonsensically without following any transformation rule. Remember: when you transform

y

ay + b

, residuals get multiplied by

a

only—the constant

b

always cancels out because it affects both observed and fitted values equally.

Question 11

In a weighted least squares problem, the objective is to minimize $\sum_{i=1}^n w_i (y_i - \beta_0 - \beta_1 x_i)^2$ where $w_i > 0$ . If the weights are $w_1 = 4, w_2 = 1, w_3 = 9$ for data points $(1, 2), (2, 5), (3, 7)$ , what is the effective design matrix $\tilde{A}$ in the transformed problem $\min \|\tilde{A}\boldsymbol{\beta} - \tilde{\mathbf{b}}\|^2$ ?

(correct answer)

Explanation: When you encounter weighted least squares problems, the key insight is transforming the weighted problem into a standard least squares form by incorporating the square roots of the weights into both the design matrix and response vector. Starting with the original design matrix for the linear model

y = \beta_0 + \beta_1 x

, you have:

To transform this into the standard form

\min \|\tilde{A}\boldsymbol{\beta} - \tilde{\mathbf{b}}\|^2

, you multiply each row by

\sqrt{w_i}

. With weights

w_1 = 4, w_2 = 1, w_3 = 9

, the square roots are

\sqrt{4} = 2, \sqrt{1} = 1, \sqrt{9} = 3

. This gives you:

This matches answer choice B. Choice A incorrectly uses

\sqrt{w_3} = 9

instead of

3

in the last row. Choice C uses the original weights instead of their square roots (

w_1 = 4

instead of

\sqrt{w_1} = 2

). Choice D appears to use

2\sqrt{w_1} = 8

in the first row, suggesting confusion about how to apply the transformation. Remember: in weighted least squares transformations, always use the square root of the weights to scale each row of both the design matrix and response vector. This converts the weighted problem into an equivalent unweighted problem that preserves the underlying linear algebra structure.

Question 12

A researcher collects data points $(1, 2), (2, 5), (3, 7), (4, 11)$ and wants to fit a line $y = ax + b$ using least squares. If the normal equations yield the system $\begin{bmatrix} 30 & 10 \\ 10 & 4 \end{bmatrix} \begin{bmatrix} a \\ b \end{bmatrix} = \begin{bmatrix} 91 \\ 25 \end{bmatrix}$ , what is the slope $a$ of the least squares line?

$a = 2.9$ (correct answer)
$a = 3.1$
$a = 2.5$
$a = 3.3$

Explanation: Solving the normal equations: From the second equation,

10a + 4b = 25

, so

b = \frac{25 - 10a}{4}

. Substituting into the first equation:

30a + 10 \cdot \frac{25 - 10a}{4} = 91

. This gives

30a + \frac{250 - 100a}{4} = 91

, which simplifies to

30a + 62.5 - 25a = 91

, so

5a = 28.5

and

a = 2.9

. Choice B results from arithmetic errors in fraction manipulation. Choice C comes from incorrectly using

b = \frac{25 - 10a}{10}

instead of dividing by 4. Choice D results from sign errors in the substitution.

Question 13

A dataset has design matrix $A$ and response vector $\mathbf{b}$ . If $\mathbf{b}$ lies in the column space of $A$ , and the least squares solution is $\hat{\mathbf{x}} = (A^T A)^{-1} A^T \mathbf{b}$ , what is the relationship between the residual vector $\mathbf{r} = \mathbf{b} - A\hat{\mathbf{x}}$ and the fitted values $A\hat{\mathbf{x}}$ ?

$\mathbf{r} = \mathbf{0}$ and $A\hat{\mathbf{x}} = \mathbf{b}$ , since $\mathbf{b}$ is exactly representable in the column space (correct answer)
$\mathbf{r} \perp A\hat{\mathbf{x}}$ and $\|\mathbf{r}\|^2 + \|A\hat{\mathbf{x}}\|^2 = \|\mathbf{b}\|^2$ , following the Pythagorean theorem
$\mathbf{r}$ is parallel to $A\hat{\mathbf{x}}$ since both are linear combinations of the columns of $A$
$\mathbf{r} \cdot A\hat{\mathbf{x}} = \frac{1}{2}\|\mathbf{b}\|^2$ due to the projection properties of least squares fitting

Explanation: If

\mathbf{b}

lies in the column space of

A

, then there exists an exact solution

\mathbf{x}_0

such that

A\mathbf{x}_0 = \mathbf{b}

. The least squares solution

\hat{\mathbf{x}} = (A^T A)^{-1} A^T \mathbf{b}

will find this exact solution, making

A\hat{\mathbf{x}} = \mathbf{b}

and thus

\mathbf{r} = \mathbf{b} - A\hat{\mathbf{x}} = \mathbf{0}

. Choice B describes the general orthogonality property but doesn't account for the special case where

\mathbf{b}

is in the column space. Choice C is incorrect because

\mathbf{r}

lies in the null space of

A^T

, not the column space of

A

. Choice D gives an arbitrary relationship that doesn't follow from projection theory.

Question 14

Two researchers analyze the same dataset but use different approaches: Researcher A centers the $x$ -data by subtracting the mean before fitting $y = \beta_0 + \beta_1 x$ , while Researcher B uses the original $x$ -values. If the original data has $\bar{x} = 5$ and the least squares line using original data is $y = 2 + 0.8x$ , what line will Researcher A obtain?

$y = 2 + 0.8x_{centered}$ where $x_{centered} = x - 5$
$y = 6 + 0.8x_{centered}$ where $x_{centered} = x - 5$ (correct answer)
$y = -2 + 0.8x_{centered}$ where $x_{centered} = x - 5$
$y = 6 + 0.64x_{centered}$ where $x_{centered} = x - 5$

Explanation: When you encounter problems about data transformations in regression, the key insight is understanding how centering affects the intercept while leaving the slope unchanged. Let's work through what happens when Researcher A centers the data. The original regression line is

y = 2 + 0.8x

with

\bar{x} = 5

. When we center the x-values by subtracting the mean, we create

x_{centered} = x - 5

, so

x = x_{centered} + 5

. Substituting this into the original equation:

y = 2 + 0.8(x_{centered} + 5) = 2 + 0.8x_{centered} + 4 = 6 + 0.8x_{centered}

The slope remains 0.8 because centering is just a horizontal shift that doesn't change the relationship's steepness. The intercept becomes 6 because when

x_{centered} = 0

(meaning

x = 5

, the original mean), we need

y = 2 + 0.8(5) = 6

. Answer A incorrectly keeps the original intercept of 2, missing that centering shifts where the line crosses the y-axis. Answer C gives -2 as the intercept, which would result from subtracting rather than adding the slope adjustment. Answer D not only has the wrong intercept but also incorrectly changes the slope to 0.64, suggesting a misunderstanding that centering affects the relationship's strength. Study tip: Remember that centering data only shifts the intercept—it never changes the slope. The new intercept equals the y-value when the centered variable equals zero (i.e., when the original variable equals its mean).

Question 15

In a least squares regression with design matrix $A \in \mathbb{R}^{n \times p}$ where $n > p$ and $\text{rank}(A) = p$ , the projection matrix is $P = A(A^T A)^{-1} A^T$ . If a new data point $(\mathbf{x}_{new}, y_{new})$ is added where $\mathbf{x}_{new}$ is orthogonal to all columns of $A$ , how does this affect the original least squares solution $\hat{\boldsymbol{\beta}}$ ?

$\hat{\boldsymbol{\beta}}$ changes by a factor proportional to $\frac{n+1}{n}$ due to the increased sample size normalization
$\hat{\boldsymbol{\beta}}$ changes significantly because the new point increases the degrees of freedom in the system
$\hat{\boldsymbol{\beta}}$ becomes undefined because the augmented design matrix loses full column rank property
$\hat{\boldsymbol{\beta}}$ remains unchanged because $\mathbf{x}_{new}$ contributes zero information about the existing parameter relationships (correct answer)

Explanation: When you encounter questions about adding data points to least squares regression, focus on how the new information interacts with your existing parameter space. The key insight here is understanding what "orthogonal to all columns of

A

" means geometrically. The least squares solution

\hat{\boldsymbol{\beta}} = (A^T A)^{-1} A^T \mathbf{y}

projects the response vector

\mathbf{y}

onto the column space of

A

. When

\mathbf{x}_{new}

is orthogonal to all columns of

A

, it lies entirely outside this column space. This means

\mathbf{x}_{new}

cannot be expressed as any linear combination of the existing columns, so it provides no information about the relationships between

\mathbf{y}

and the original features. Mathematically, since

\mathbf{x}_{new} \perp \text{col}(A)

, we have

A^T \mathbf{x}_{new} = \mathbf{0}

. The new point adds a residual that's orthogonal to the original fitted values, leaving the projection onto the original column space unchanged. Therefore,

\hat{\boldsymbol{\beta}}

remains unchanged, making D correct. Option A incorrectly suggests a scaling effect from sample size, but least squares doesn't normalize by

n

. Option B misunderstands degrees of freedom—orthogonal directions don't affect existing parameter estimates. Option C is wrong because adding a linearly independent vector (which orthogonal

\mathbf{x}_{new}

is) actually increases rank, not decreases it. Study tip: Remember that in regression, only the component of new data that lies within your existing feature space affects parameter estimates. Orthogonal components contribute only to unexplained variance, not to

\hat{\boldsymbol{\beta}}

Question 16

The least-squares solution $\hat{x}$ to the system $A\vec{x} = \vec{b}$ is unique if and only if which condition is met?

The columns of $A$ are linearly independent. (correct answer)
The matrix $A$ is square and invertible.
The columns of $A$ are orthogonal.
The vector $\vec{b}$ is in the column space of $A$ .

Explanation: The least-squares solution is found by solving the normal equations

A^T A \hat{x} = A^T \vec{b}

. This system has a unique solution if and only if the matrix

A^T A

is invertible. The matrix

A^T A

is invertible if and only if the columns of

A

are linearly independent. If

A

is square and invertible (B), there is a unique exact solution, which is a special case. Orthogonal columns (C) is a sufficient condition for linear independence, but not a necessary one. If

\vec{b}

is in

\mathrm{Col}(A)

(D), an exact solution exists, but it may not be unique if the columns of

A

are linearly dependent.

Question 17

What is the slope of the least-squares regression line for the data points $(-1, 0)$ , $(0, 1)$ , $(1, 3)$ , and $(2, 4)$ ?

$0.5$
$1.33$
$1.4$ (correct answer)
$1.75$

Explanation: The model is

y = c_0 + c_1 x

. The matrix system is

A\vec{c} = \vec{b}

with

A = \begin{pmatrix} 1 & -1 \\ 1 & 0 \\ 1 & 1 \\ 1 & 2 \end{pmatrix}

and

\vec{b} = \begin{pmatrix} 0 \\ 1 \\ 3 \\ 4 \end{pmatrix}

. The normal equations are

A^T A \vec{c} = A^T \vec{b}

. We compute

A^T A = \begin{pmatrix} 4 & 2 \\ 2 & 6 \end{pmatrix}

and

A^T \vec{b} = \begin{pmatrix} 8 \\ 11 \end{pmatrix}

. We must solve

\begin{pmatrix} 4 & 2 \\ 2 & 6 \end{pmatrix} \begin{pmatrix} c_0 \\ c_1 \end{pmatrix} = \begin{pmatrix} 8 \\ 11 \end{pmatrix}

. The second row gives

2c_0 + 6c_1 = 11

. The first row gives

4c_0 + 2c_1 = 8

, or

2c_0 + c_1 = 4

. Subtracting the two equations:

(2c_0 + 6c_1) - (2c_0 + c_1) = 11 - 4

, which simplifies to

5c_1 = 7

, so

c_1 = 1.4

. The slope is

c_1 = 1.4

Question 18

Let $\hat{x}$ be a least-squares solution to an inconsistent system $A\vec{x} = \vec{b}$ . Which of the following conditions is NOT necessarily satisfied by $\hat{x}$ ?

$A^T(\vec{b} - A\hat{x}) = \vec{0}$
$A(\vec{b} - A\hat{x}) = \vec{0}$ (correct answer)
$A\hat{x} = \mathrm{proj}_{\mathrm{Col}(A)}\vec{b}$
$\| \vec{b} - A\hat{x} \| \le \| \vec{b} - A\vec{z} \|$ for all $\vec{z}$ in the domain.

Explanation: The defining property of the least-squares solution is that the error vector,

\vec{e} = \vec{b} - A\hat{x}

, is orthogonal to the column space of

A

. This orthogonality condition is expressed as

A^T\vec{e} = \vec{0}

, which is choice A. Choice D is the definition of minimizing the residual norm. Choice C is the geometric interpretation of the result. Choice B,

A(\vec{b} - A\hat{x}) = \vec{0}

, states that the error vector is in the null space of

A

. This is not generally true; the error vector is in the null space of

A^T

(the orthogonal complement of the column space of

A

Question 19

If the least-squares error for the system $A\vec{x} = \vec{b}$ is zero, what can be definitively concluded?

The matrix $A$ has a trivial null space.
The solution $\hat{x}$ must be the zero vector.
The vector $\vec{b}$ is in the column space of $A$ . (correct answer)
The columns of $A$ form an orthonormal set.

Explanation: The least-squares error is given by

\| \vec{b} - A\hat{x} \|

. If this error is zero, it means

\vec{b} - A\hat{x} = \vec{0}

, or

A\hat{x} = \vec{b}

. This implies that there exists a vector

\hat{x}

such that

\vec{b}

can be written as a linear combination of the columns of

A

. By definition, this means

\vec{b}

is in the column space of

A

, and the system was consistent from the start. A non-trivial null space (A) could still exist if there are multiple solutions. The solution

\hat{x}

(B) is only zero if

\vec{b}

is zero (and columns of A are LI). The columns of

A

(D) do not need to be orthonormal.

Question 20

Let $\hat{x}$ be the least-squares solution to the system $A\vec{x} = \vec{b}$ . The vector $A\hat{x}$ represents which of the following?

The orthogonal projection of $\vec{b}$ onto the column space of $A$ . (correct answer)
The orthogonal projection of $\vec{b}$ onto the row space of $A$ .
The error vector, which is orthogonal to the column space of $A$ .
The component of the solution $\hat{x}$ that lies in the column space of $A$ .

Explanation: The least-squares solution

\hat{x}

is found such that

A\hat{x}

is the vector in the column space of

A

(denoted

\mathrm{Col}(A)

) that is closest to

\vec{b}

. This closest vector is the orthogonal projection of

\vec{b}

onto

\mathrm{Col}(A)

. The error vector is

\vec{b} - A\hat{x}

, not

A\hat{x}

. The row space is not the correct subspace for the projection of

\vec{b}

. The solution vector

\hat{x}

is in the domain

\mathbb{R}^n

, not the codomain where

\mathrm{Col}(A)

resides.