Linear Algebra Quiz: Normal Equations

20 questions · exam conditions

0:00

Normal EquationsQuestion 1 of 20

Consider the problem of finding the least-squares solution to the system of equations: $-$ x + y = 3\quad x + y = 0\quad y = 4\ Which of the following represents the corresponding normal equations $A^T A \hat{\mathbf{x}} = A^T \mathbf{b}$ ?

\begin{pmatrix} 2 & 0 \\ 0 & 3 \end{pmatrix} \hat{\mathbf{x}} = \begin{pmatrix} -3 \\ 7 \end{pmatrix}

\begin{pmatrix} 2 & 0 \\ 0 & 3 \end{pmatrix} \hat{\mathbf{x}} = \begin{pmatrix} 3 \\ 7 \end{pmatrix}

\begin{pmatrix} 3 & 1 \\ 1 & 2 \end{pmatrix} \hat{\mathbf{x}} = \begin{pmatrix} -3 \\ 7 \end{pmatrix}

\begin{pmatrix} 2 & 1 & 1 \\ 1 & 2 & 1 \\ 1 & 1 & 1 \end{pmatrix} \hat{\mathbf{x}} = \begin{pmatrix} 3 \\ 0 \\ 4 \end{pmatrix}

← Back to quizzes

Linear Algebra Quiz

Linear Algebra Quiz: Normal Equations

Practice Normal Equations in Linear Algebra with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on Normal Equations, giving you a quick way to practice the rules, question types, and explanations that matter most for Linear Algebra.

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

$\begin{pmatrix} 2 & 0 \\ 0 & 3 \end{pmatrix} \hat{\mathbf{x}} = \begin{pmatrix} -3 \\ 7 \end{pmatrix}$ (correct answer)
$\begin{pmatrix} 2 & 0 \\ 0 & 3 \end{pmatrix} \hat{\mathbf{x}} = \begin{pmatrix} 3 \\ 7 \end{pmatrix}$
$\begin{pmatrix} 3 & 1 \\ 1 & 2 \end{pmatrix} \hat{\mathbf{x}} = \begin{pmatrix} -3 \\ 7 \end{pmatrix}$
$\begin{pmatrix} 2 & 1 & 1 \\ 1 & 2 & 1 \\ 1 & 1 & 1 \end{pmatrix} \hat{\mathbf{x}} = \begin{pmatrix} 3 \\ 0 \\ 4 \end{pmatrix}$

Explanation: The system can be written as

A\mathbf{x}=\mathbf{b}

where

A = \begin{pmatrix} -1 & 1 \\ 1 & 1 \\ 0 & 1 \end{pmatrix}

\mathbf{x} = \begin{pmatrix} x \\ y \end{pmatrix}

, and

\mathbf{b} = \begin{pmatrix} 3 \\ 0 \\ 4 \end{pmatrix}

. We compute

A^T A

and

A^T \mathbf{b}

A^T = \begin{pmatrix} -1 & 1 & 0 \\ 1 & 1 & 1 \end{pmatrix}

A^T A = \begin{pmatrix} -1 & 1 & 0 \\ 1 & 1 & 1 \end{pmatrix} \begin{pmatrix} -1 & 1 \\ 1 & 1 \\ 0 & 1 \end{pmatrix} = \begin{pmatrix} 2 & 0 \\ 0 & 3 \end{pmatrix}

A^T \mathbf{b} = \begin{pmatrix} -1 & 1 & 0 \\ 1 & 1 & 1 \end{pmatrix} \begin{pmatrix} 3 \\ 0 \\ 4 \end{pmatrix} = \begin{pmatrix} -3 \\ 7 \end{pmatrix}

. Thus, the normal equations are

\begin{pmatrix} 2 & 0 \\ 0 & 3 \end{pmatrix} \hat{\mathbf{x}} = \begin{pmatrix} -3 \\ 7 \end{pmatrix}

Question 2

Let $\hat{\mathbf{x}}$ be the least-squares solution to $A\mathbf{x} = \mathbf{b}$ , and let $W = \mathrm{Col}(A)$ . Which of the following vectors must lie in the orthogonal complement of $W$ , denoted $W^\perp$ ?

The vector $\mathbf{b}$ being approximated.
The least-squares solution vector $\hat{\mathbf{x}}$ .
The projection vector $A\hat{\mathbf{x}}$ .
The residual vector $\mathbf{b} - A\hat{\mathbf{x}}$ . (correct answer)

Explanation: The fundamental principle of least squares is that the error vector (or residual vector)

\mathbf{b} - A\hat{\mathbf{x}}

is orthogonal to the subspace onto which we are projecting, which is the column space

W

. Therefore,

\mathbf{b} - A\hat{\mathbf{x}}

is in

W^\perp

. The vector

A\hat{\mathbf{x}}

is in

W

itself. The vector

\hat{\mathbf{x}}

is in

\mathbb{R}^n

(domain of the transformation), not necessarily related orthogonally to

W

. The vector

\mathbf{b}

is generally not in

W^\perp

Question 3

The system $A^T A \mathbf{x} = A^T \mathbf{b}$ is called the 'normal equations' because its solution $\hat{\mathbf{x}}$ ensures a specific orthogonality condition, from which the name is derived. What is this condition?

The solution vector $\hat{\mathbf{x}}$ is normal (orthogonal) to the vector $\mathbf{b}$ .
The columns of the matrix $A$ are required to be normal (orthogonal) to each other.
The residual vector $(\mathbf{b} - A\hat{\mathbf{x}})$ is normal (orthogonal) to the column space of $A$ . (correct answer)
The projection of $\mathbf{b}$ onto $\mathrm{Col}(A)$ is normal (orthogonal) to $\mathbf{b}$ itself.

Explanation: The term 'normal' is synonymous with 'orthogonal'. The normal equations are derived by enforcing the condition that the residual vector

\mathbf{b} - A\hat{\mathbf{x}}

must be orthogonal to every column of

A

, and thus to the entire column space of

A

. This is the core geometric principle. The other options describe conditions that are not generally true or are special cases not central to the name's origin.

Question 4

Suppose the system $A\mathbf{x} = \mathbf{b}$ is consistent, which implies that $\mathbf{b}$ is already in the column space of $A$ . If the normal equations are used to find a least-squares solution $\hat{\mathbf{x}}$ , what is the value of the least-squares error, defined as $\|\mathbf{b} - A\hat{\mathbf{x}}\|$ ?

$0$ (correct answer)
$1$
$\|\mathbf{b}\|$
It cannot be determined without knowing the matrix $A$ .

Explanation: If the system

A\mathbf{x} = \mathbf{b}

is consistent, it means there exists an exact solution. The least-squares method finds the vector in

\mathrm{Col}(A)

closest to

\mathbf{b}

. Since

\mathbf{b}

is already in

\mathrm{Col}(A)

, the closest vector is

\mathbf{b}

itself. Therefore, the projection of

\mathbf{b}

\mathbf{b}

, meaning

A\hat{\mathbf{x}} = \mathbf{b}

. The error

\|\mathbf{b} - A\hat{\mathbf{x}}\| = \|\mathbf{b} - \mathbf{b}\| = 0

Question 5

To find the best-fit line of the form $y = c_0 + c_1 x$ for the data points $(0, 1)$ , $(1, 3)$ , and $(2, 4)$ , one must find the least-squares solution to the system $A\mathbf{c} = \mathbf{y}$ . What are the normal equations $A^T A \mathbf{c} = A^T \mathbf{y}$ for this problem?

$\begin{pmatrix} 3 & 3 \\ 3 & 5 \end{pmatrix} \mathbf{c} = \begin{pmatrix} 8 \\ 14 \end{pmatrix}$
$\begin{pmatrix} 1 & 1 & 1 \\ 0 & 1 & 2 \end{pmatrix} \mathbf{c} = \begin{pmatrix} 1 \\ 3 \\ 4 \end{pmatrix}$
$\begin{pmatrix} 3 & 5 \\ 5 & 9 \end{pmatrix} \mathbf{c} = \begin{pmatrix} 8 \\ 11 \end{pmatrix}$
$\begin{pmatrix} 3 & 3 \\ 3 & 5 \end{pmatrix} \mathbf{c} = \begin{pmatrix} 8 \\ 11 \end{pmatrix}$ (correct answer)

Explanation: When you encounter a least-squares problem, you're solving an overdetermined system where you have more equations than unknowns. The key is setting up the correct matrix equation

A\mathbf{c} = \mathbf{y}

and then finding the normal equations

A^T A \mathbf{c} = A^T \mathbf{y}

. For the line

y = c_0 + c_1 x

through points

(0,1)

(1,3)

, and

(2,4)

, you need matrix

A

where each row corresponds to a data point. The first column represents the coefficient of

c_0

(always 1), and the second column represents the coefficient of

c_1

(the

x

-value):

Computing

A^T A

Computing

A^T \mathbf{y}

This confirms answer D is correct. Option A has the wrong

A^T \mathbf{y}

vector (14 instead of 11). Option B shows the original system

A\mathbf{c} = \mathbf{y}

, not the normal equations. Option C has incorrect entries in

A^T A

- the computation errors likely come from mishandling the matrix multiplication. Study tip: Always double-check your matrix multiplications when computing normal equations. Set up

A

systematically: first column all 1's for the constant term, subsequent columns for the

x

-coefficients.

Question 6

Let $P = A(A^T A)^{-1} A^T$ be the projection matrix that projects vectors in $\mathbb{R}^m$ onto the column space of an $m \times n$ matrix $A$ . Which of the following expressions must evaluate to the $n \times 1$ zero vector for any vector $\mathbf{b}$ in $\mathbb{R}^m$ ?

$A^T P \mathbf{b}$
$A^T (\mathbf{b} - P\mathbf{b})$ (correct answer)
$P(\mathbf{b} - A^T \mathbf{b})$
$A(P\mathbf{b} - \mathbf{b})$

Explanation: The vector

P\mathbf{b}

is the projection of

\mathbf{b}

onto

\mathrm{Col}(A)

, which we denote as

\hat{\mathbf{b}}

. The vector

\mathbf{b} - P\mathbf{b}

is the error vector

\mathbf{e}

, which is orthogonal to

\mathrm{Col}(A)

. The condition that

\mathbf{e}

is orthogonal to

\mathrm{Col}(A)

is precisely that it is orthogonal to every column of

A

. This can be written as

A^T \mathbf{e} = \mathbf{0}

, which is

A^T (\mathbf{b} - P\mathbf{b}) = \mathbf{0}

Question 7

Let $W$ be a subspace of $\mathbb{R}^m$ and let $\mathbf{b}$ be a vector in $\mathbb{R}^m$ . The least-squares solution $\hat{\mathbf{x}}$ to $A\mathbf{x}=\mathbf{b}$ (where $W=\mathrm{Col}(A)$ ) minimizes the distance $\|\mathbf{b} - \mathbf{w}\|$ for all $\mathbf{w} \in W$ . Which expression represents the square of this minimum distance?

$\|\mathbf{b}\|^2 + \|A(A^T A)^{-1} A^T \mathbf{b}\|^2$
$\|A(A^T A)^{-1} A^T \mathbf{b}\|^2$
$\|(A^T A)^{-1} A^T \mathbf{b}\|^2$
$\|\mathbf{b} - A(A^T A)^{-1} A^T \mathbf{b}\|^2$ (correct answer)

Explanation: When you encounter least-squares problems, you're dealing with finding the best approximation to a vector

\mathbf{b}

within a subspace

W

. The key insight is that the minimum distance occurs when we project

\mathbf{b}

onto

W

, and the distance we want is from

\mathbf{b}

to this projection. The projection of

\mathbf{b}

onto the column space of

A

is given by

P\mathbf{b} = A(A^T A)^{-1} A^T \mathbf{b}

, where

P = A(A^T A)^{-1} A^T

is the projection matrix. The minimum distance is the length of the error vector:

\mathbf{b} - P\mathbf{b} = \mathbf{b} - A(A^T A)^{-1} A^T \mathbf{b}

. Therefore, the square of this minimum distance is

\|\mathbf{b} - A(A^T A)^{-1} A^T \mathbf{b}\|^2

, which is answer D. Let's examine why the other options are incorrect. Option A represents

\|\mathbf{b}\|^2

plus the square of the projection length, which doesn't correspond to any meaningful geometric quantity in this context. Option B gives you the square of the length of the projection itself, not the distance from

\mathbf{b}

to the projection. Option C represents the square of the length of the least-squares solution vector

\hat{\mathbf{x}} = (A^T A)^{-1} A^T \mathbf{b}

, which lives in the domain space, not the distance we're measuring. Remember: in least-squares problems, you're always measuring the distance between the original vector and its projection onto the subspace. The answer will involve the difference

\mathbf{b} - \text{projection}

, not just the projection itself.

Question 8

Let $A$ be an $m \times n$ matrix and $\mathbf{b}$ be a vector in $\mathbb{R}^m$ . If $\hat{\mathbf{x}}$ is the least-squares solution to the system $A\mathbf{x} = \mathbf{b}$ , which statement best describes the geometric relationship involving the error vector $\mathbf{e} = \mathbf{b} - A\hat{\mathbf{x}}$ ?

The error vector $\mathbf{e}$ is orthogonal to the column space of $A$ . (correct answer)
The error vector $\mathbf{e}$ is parallel to the projection of $\mathbf{b}$ onto the column space of $A$ .
The error vector $\mathbf{e}$ is orthogonal to the solution vector $\hat{\mathbf{x}}$ .
The error vector $\mathbf{e}$ is an element of the column space of $A$ .

Explanation: The normal equations

A^T A \hat{\mathbf{x}} = A^T \mathbf{b}

are derived from the orthogonality condition

A^T(\mathbf{b} - A\hat{\mathbf{x}}) = \mathbf{0}

. This condition means that the error vector

\mathbf{e} = \mathbf{b} - A\hat{\mathbf{x}}

is orthogonal to every column of

A

. Therefore, the error vector is orthogonal to the entire column space of

A

Question 9

The normal equations $A^T A \hat{\mathbf{x}} = A^T \mathbf{b}$ produce the set of all least-squares solutions. Under what geometric condition will there be infinitely many least-squares solutions $\hat{\mathbf{x}}$ ?

When the vector $\mathbf{b}$ is orthogonal to the column space of $A$ .
When the columns of the matrix $A$ are linearly dependent. (correct answer)
When the number of rows in $A$ is greater than the number of its columns.
When the system $A\mathbf{x} = \mathbf{b}$ is consistent (i.e., has an exact solution).

Explanation: The set of least-squares solutions is unique if and only if the matrix

A^T A

is invertible. This occurs if and only if the columns of

A

are linearly independent. If the columns of

A

are linearly dependent, then

A^T A

is singular, and the system

A^T A \hat{\mathbf{x}} = A^T \mathbf{b}

will have infinitely many solutions. Geometrically, this means that a vector in the column space can be formed as a linear combination of the columns in more than one way.

Question 10

Let $W$ be the column space of a matrix $A$ . For a vector $\mathbf{b}$ , the least-squares solution $\hat{\mathbf{x}}$ to $A\mathbf{x} = \mathbf{b}$ is found. What is the geometric meaning of the vector $\hat{\mathbf{b}} = A\hat{\mathbf{x}}$ ?

$\hat{\mathbf{b}}$ is the component of $\mathbf{b}$ that is orthogonal to the column space $W$ .
$\hat{\mathbf{b}}$ is the orthogonal projection of $\mathbf{b}$ onto the column space $W$ . (correct answer)
$\hat{\mathbf{b}}$ is the orthogonal projection of $\mathbf{b}$ onto the null space of $A$ .
$\hat{\mathbf{b}}$ is a vector in the row space of $A$ that is closest to $\mathbf{b}$ .

Explanation: By definition, the vector

A\hat{\mathbf{x}}

is the orthogonal projection of

\mathbf{b}

onto the column space of

A

. This vector is the point in

W

that is closest to

\mathbf{b}

. The vector described in (A) is the error vector

\mathbf{b} - \hat{\mathbf{b}}

. The subspaces in (C) and (D) are incorrect.

Question 11

The normal equations $A^T A \hat{\mathbf{x}} = A^T \mathbf{b}$ are used to find the least-squares solution to an inconsistent system $A\mathbf{x} = \mathbf{b}$ . A unique least-squares solution $\hat{\mathbf{x}}$ is guaranteed to exist if and only if:

the matrix $A$ is a square and invertible matrix.
the columns of the matrix $A$ are orthogonal to each other.
the columns of the matrix $A$ are linearly independent. (correct answer)
the vector $\mathbf{b}$ is in the column space of $A$ .

Explanation: A unique least-squares solution exists if and only if the matrix

A^T A

is invertible. The matrix

A^T A

is invertible if and only if the columns of

A

are linearly independent. Choice (A) and (B) are sufficient conditions but not necessary. Choice (D) means the system

A\mathbf{x}=\mathbf{b}

is consistent and has an exact solution, but it does not guarantee the solution is unique unless the columns of

A

are linearly independent.

Question 12

Given the matrix equation $A\mathbf{x} = \mathbf{b}$ where $A$ is $m \times n$ with $m > n$ , suppose the normal equations yield $\hat{\mathbf{x}} = \begin{pmatrix} 2 \\ -1 \end{pmatrix}$ . If the residual vector is $\mathbf{r} = \mathbf{b} - A\hat{\mathbf{x}} = \begin{pmatrix} 1 \\ 0 \\ -2 \end{pmatrix}$ , what can be concluded about the relationship between $\mathbf{r}$ and the columns of $A$ ?

The residual $\mathbf{r}$ is orthogonal to every column of $A$ because $A^T\mathbf{r} = \mathbf{0}$ is satisfied by the normal equations (correct answer)
The residual $\mathbf{r}$ is parallel to the first column of $A$ since the first component of $\hat{\mathbf{x}}$ is positive
The residual $\mathbf{r}$ lies in the column space of $A$ because it can be written as a linear combination of $A$ 's columns
The residual $\mathbf{r}$ is orthogonal to $\mathbf{b}$ but not necessarily to the individual columns of $A$

Explanation: The normal equations

A^TA\hat{\mathbf{x}} = A^T\mathbf{b}

can be rewritten as

A^T(A\hat{\mathbf{x}} - \mathbf{b}) = \mathbf{0}

, which means

A^T\mathbf{r} = \mathbf{0}

. This implies that

\mathbf{r}

is orthogonal to every column of

A

. Choice B is incorrect because the sign of components in

\hat{\mathbf{x}}

doesn't determine parallelism with columns. Choice C is wrong because

\mathbf{r}

lies in the orthogonal complement of the column space. Choice D is incorrect because orthogonality to

\mathbf{b}

is not guaranteed, and orthogonality to columns is guaranteed.

Question 13

Consider fitting a line $y = ax + b$ to the data points $(0,1)$ , $(1,3)$ , and $(2,4)$ using least squares. The normal equations for this problem take the form $$ \begin{pmatrix} \sum x_i^2 & \sum x_i \ \sum x_i & n \end{pmatrix}\begin{pmatrix} a \ b \end{pmatrix} = \begin{pmatrix} \sum x_i y_i \ \sum y_i \end{pmatrix}

$\begin{pmatrix} 5 & 3 \\ 3 & 3 \end{pmatrix}$ (correct answer)
$\begin{pmatrix} 3 & 3 \\ 3 & 5 \end{pmatrix}$
$\begin{pmatrix} 5 & 3 \\ 3 & 8 \end{pmatrix}$
$\begin{pmatrix} 8 & 3 \\ 3 & 3 \end{pmatrix}$

Explanation: For the data points

(0,1)

(1,3)

(2,4)

, we have

n = 3

points. Calculate:

\sum x_i^2 = 0^2 + 1^2 + 2^2 = 5

\sum x_i = 0 + 1 + 2 = 3

, and

n = 3

. The coefficient matrix is

\begin{pmatrix} 5 & 3 \\ 3 & 3 \end{pmatrix}

. Choice B swaps the positions of

\sum x_i^2

and

n

. Choice C incorrectly uses

\sum y_i = 8

in place of

n = 3

. Choice D incorrectly uses

\sum y_i = 8

in the (1,1) position instead of

\sum x_i^2 = 5

Question 14

Consider the quadratic form $Q(\mathbf{x}) = ||A\mathbf{x} - \mathbf{b}||^2$ where $A$ is $m \times n$ with linearly independent columns. The normal equations arise from setting $\nabla Q(\mathbf{x}) = \mathbf{0}$ . Which expression correctly represents the gradient $\nabla Q(\mathbf{x})$ ?

$\nabla Q(\mathbf{x}) = 2A(A^T\mathbf{x} - \mathbf{b})$
$\nabla Q(\mathbf{x}) = A^T(A\mathbf{x} - \mathbf{b})$
$\nabla Q(\mathbf{x}) = 2A^T(A\mathbf{x} - \mathbf{b})$ (correct answer)
$\nabla Q(\mathbf{x}) = (A^TA)\mathbf{x} - A^T\mathbf{b}$

Explanation: When you encounter quadratic forms involving matrix norms like

Q(\mathbf{x}) = ||A\mathbf{x} - \mathbf{b}||^2

, you're dealing with least squares optimization. The key insight is recognizing this as a composite function that requires the chain rule for differentiation. To find

\nabla Q(\mathbf{x})

, first expand the squared norm:

Q(\mathbf{x}) = (A\mathbf{x} - \mathbf{b})^T(A\mathbf{x} - \mathbf{b})

. This gives you

Q(\mathbf{x}) = \mathbf{x}^TA^TA\mathbf{x} - 2\mathbf{b}^TA\mathbf{x} + \mathbf{b}^T\mathbf{b}

. Taking the gradient term by term: the gradient of

\mathbf{x}^TA^TA\mathbf{x}

2A^TA\mathbf{x}

, the gradient of

-2\mathbf{b}^TA\mathbf{x}

-2A^T\mathbf{b}

, and the constant term disappears. This yields

\nabla Q(\mathbf{x}) = 2A^TA\mathbf{x} - 2A^T\mathbf{b} = 2A^T(A\mathbf{x} - \mathbf{b})

. Choice C is correct with this factor of 2 included. Choice A incorrectly places

A

instead of

A^T

in front, which would give the wrong dimensions. Choice B has the right structure but missing the factor of 2 that comes from differentiating the squared term. Choice D shows the expanded form before factoring, which is mathematically equivalent to C but doesn't match the standard factored form asked for. Remember: when differentiating squared norms, always expect a factor of 2, and the transpose of the coefficient matrix (

A^T

) appears in the gradient due to the chain rule.

Question 15

In a least squares problem, suppose $A = QR$ where $Q$ has orthonormal columns and $R$ is upper triangular and invertible. How do the normal equations $A^TA\mathbf{x} = A^T\mathbf{b}$ simplify when expressed in terms of the QR decomposition?

$QR\mathbf{x} = Q^T\mathbf{b}$
$R^TR\mathbf{x} = R^TQ^T\mathbf{b}$
$R\mathbf{x} = Q^T\mathbf{b}$ (correct answer)
$Q^TQ R\mathbf{x} = R^T\mathbf{b}$

Explanation: When you encounter QR decomposition in least squares problems, you're looking at a powerful way to simplify the normal equations by exploiting the orthogonal structure of matrix

Q

. Starting with the normal equations

A^TA\mathbf{x} = A^T\mathbf{b}

and substituting

A = QR

, you get:

(QR)^T(QR)\mathbf{x} = (QR)^T\mathbf{b}

This expands to:

R^TQ^TQR\mathbf{x} = R^TQ^T\mathbf{b}

Here's the key insight: since

Q

has orthonormal columns,

Q^TQ = I

(the identity matrix). This dramatically simplifies the equation to:

R^TR\mathbf{x} = R^TQ^T\mathbf{b}

Since

R

is invertible, you can multiply both sides by

(R^T)^{-1}

to get:

R\mathbf{x} = Q^T\mathbf{b}

This is answer C. Looking at the wrong choices: A)

QR\mathbf{x} = Q^T\mathbf{b}

incorrectly keeps the original

A = QR

without accounting for the

A^TA

structure. B)

R^TR\mathbf{x} = R^TQ^T\mathbf{b}

stops one step short of the full simplification—it's correct but not simplified. D)

Q^TQR\mathbf{x} = R^T\mathbf{b}

has the wrong right-hand side and fails to use the orthonormality property. Study tip: Remember that

Q^TQ = I

for orthonormal columns is the magic that makes QR decomposition so useful—it eliminates the problematic

A^TA

multiplication that can cause numerical instability.

Question 16

Consider two overdetermined systems: System 1 has matrix $A_1$ and right-hand side $\mathbf{b}_1$ , while System 2 has matrix $A_2 = A_1$ and right-hand side $\mathbf{b}_2 = 2\mathbf{b}_1$ . If $\hat{\mathbf{x}}_1$ and $\hat{\mathbf{x}}_2$ are the respective least squares solutions, what is the relationship between the residual norms $||\mathbf{b}_1 - A_1\hat{\mathbf{x}}_1||$ and $||\mathbf{b}_2 - A_2\hat{\mathbf{x}}_2||$ ?

$||\mathbf{b}_2 - A_2\hat{\mathbf{x}}_2|| = 4||\mathbf{b}_1 - A_1\hat{\mathbf{x}}_1||$
$||\mathbf{b}_2 - A_2\hat{\mathbf{x}}_2|| = 2||\mathbf{b}_1 - A_1\hat{\mathbf{x}}_1||$ (correct answer)
$||\mathbf{b}_2 - A_2\hat{\mathbf{x}}_2|| = ||\mathbf{b}_1 - A_1\hat{\mathbf{x}}_1||$
$||\mathbf{b}_2 - A_2\hat{\mathbf{x}}_2|| = \sqrt{2}||\mathbf{b}_1 - A_1\hat{\mathbf{x}}_1||$

Explanation: When you encounter problems comparing least squares solutions with scaled right-hand sides, the key insight is that least squares solutions and residuals both scale linearly with the right-hand side vector. For System 1:

A_1\hat{\mathbf{x}}_1 \approx \mathbf{b}_1

, and for System 2:

A_2\hat{\mathbf{x}}_2 \approx \mathbf{b}_2

where

A_2 = A_1

and

\mathbf{b}_2 = 2\mathbf{b}_1

. Since the matrix is identical in both systems, but the right-hand side is scaled by 2, the least squares solution also scales by the same factor:

\hat{\mathbf{x}}_2 = 2\hat{\mathbf{x}}_1

. You can verify this because if

\hat{\mathbf{x}}_1

minimizes

||A_1\mathbf{x} - \mathbf{b}_1||^2

, then

2\hat{\mathbf{x}}_1

minimizes

||A_1\mathbf{x} - 2\mathbf{b}_1||^2

. The residual for System 2 becomes:

\mathbf{b}_2 - A_2\hat{\mathbf{x}}_2 = 2\mathbf{b}_1 - A_1(2\hat{\mathbf{x}}_1) = 2(\mathbf{b}_1 - A_1\hat{\mathbf{x}}_1)

Taking norms:

||\mathbf{b}_2 - A_2\hat{\mathbf{x}}_2|| = ||2(\mathbf{b}_1 - A_1\hat{\mathbf{x}}_1)|| = 2||\mathbf{b}_1 - A_1\hat{\mathbf{x}}_1||

This confirms answer B is correct. A incorrectly squares the scaling factor (

2^2 = 4

), confusing linear scaling with quadratic relationships. C suggests no relationship, ignoring the scaling entirely. D uses

\sqrt{2}

, which would arise from some geometric relationship but doesn't apply here. Study tip: Remember that least squares solutions scale linearly with the right-hand side when the coefficient matrix stays the same. Both the solution and residual scale by the same factor as the right-hand side.

Question 17

A researcher sets up normal equations to solve an overdetermined linear system, but discovers that the coefficient matrix $A^TA$ is singular. Given that the original matrix $A$ is $4 \times 3$ , what is the most likely explanation for this singularity, and what does it imply about the solution?

The system is overdetermined with $m > n$ ; singular $A^TA$ indicates the original system has a unique exact solution
The matrix $A$ has more rows than columns; the normal equations are inconsistent and have no solution
The vector $\mathbf{b}$ lies outside the column space of $A$ ; the least squares problem has no meaningful solution
The columns of $A$ are linearly dependent; the normal equations have infinitely many solutions lying on an affine subspace (correct answer)

Explanation: When you encounter normal equations and a singular

A^TA

matrix, you're dealing with fundamental concepts about linear independence and the least squares solution structure. The key insight is understanding what makes

A^TA

singular. Since

A

4 \times 3

, we have more equations than unknowns (overdetermined system). The matrix

A^TA

becomes singular precisely when the columns of

A

are linearly dependent. This happens because

A^TA

inherits the rank deficiency from

A

- if

\text{rank}(A) < 3

, then

\text{rank}(A^TA) < 3

, making it non-invertible. When

A^TA

is singular, the normal equations

A^TA\mathbf{x} = A^T\mathbf{b}

have infinitely many solutions forming an affine subspace (a translated version of the null space). This means multiple parameter vectors can minimize the least squares objective function equally well. Choice A incorrectly suggests a unique exact solution exists - but overdetermined systems rarely have exact solutions, and singularity indicates non-uniqueness. Choice B wrongly claims the normal equations are inconsistent; they're actually always consistent since

A^T\mathbf{b}

lies in the column space of

A^TA

. Choice C misunderstands the relationship between

\mathbf{b}

and solvability - even when

\mathbf{b}

isn't in

A

's column space, least squares solutions still exist and are meaningful. Study tip: Remember that

A^TA

singular always means linearly dependent columns in

A

, which leads to infinitely many least squares solutions. This is a structural property of the matrix, not the right-hand side vector.

Question 18

For an inconsistent linear system $A\mathbf{x} = \mathbf{b}$ , the normal equations $A^TA\mathbf{x} = A^T\mathbf{b}$ always have a solution. If $A$ has linearly independent columns, which statement best explains why the normal equations are consistent while the original system is not?

The matrix $A^TA$ is always invertible when $A$ has linearly independent columns, guaranteeing a unique solution regardless of $\mathbf{b}$ (correct answer)
The vector $A^T\mathbf{b}$ automatically lies in the column space of $A^TA$ because $A^TA$ has the same column space as $A^T$
The normal equations project the inconsistent system onto a consistent subspace by ensuring the residual is orthogonal to the column space
The transformation $A^T$ maps any vector $\mathbf{b}$ into the row space of $A$ , which is always compatible with the column space of $A^TA$

Explanation: When

A

has linearly independent columns,

A^TA

is invertible (positive definite), so

A^TA\mathbf{x} = A^T\mathbf{b}

has the unique solution

\mathbf{x} = (A^TA)^{-1}A^T\mathbf{b}

for any

\mathbf{b}

. Choice B is incorrect because

A^TA

has the same column space as

A^T

, but this doesn't automatically guarantee

A^T\mathbf{b}

lies in it. Choice C describes what the solution does but not why the equations are consistent. Choice D incorrectly describes the relationship between row and column spaces.

Question 19

For the overdetermined system $A\mathbf{x} = \mathbf{b}$ , let $P$ be the orthogonal projection matrix onto the column space of $A$ . If the normal equations have solution $\hat{\mathbf{x}}$ , which of the following expressions correctly represents the squared norm of the residual $||\mathbf{b} - A\hat{\mathbf{x}}||^2$ ?

$\mathbf{b}^T A \hat{\mathbf{x}} - ||A\hat{\mathbf{x}}||^2$
$||P\mathbf{b}||^2 - ||\mathbf{b}||^2$
$||\mathbf{b}||^2 - \mathbf{b}^T A \hat{\mathbf{x}}$
$||\mathbf{b}||^2 - ||P\mathbf{b}||^2$ (correct answer)

Explanation: When you encounter questions about least squares and projection matrices, remember that the residual represents the "leftover" part of

\mathbf{b}

that cannot be expressed in the column space of

A

. The key insight is understanding what happens geometrically. The vector

A\hat{\mathbf{x}}

is the orthogonal projection of

\mathbf{b}

onto the column space of

A

, which we can write as

P\mathbf{b}

. The residual

\mathbf{b} - A\hat{\mathbf{x}} = \mathbf{b} - P\mathbf{b}

is orthogonal to the column space. By the Pythagorean theorem in this orthogonal decomposition:

||\mathbf{b}||^2 = ||P\mathbf{b}||^2 + ||\mathbf{b} - P\mathbf{b}||^2

Rearranging gives us

||\mathbf{b} - A\hat{\mathbf{x}}||^2 = ||\mathbf{b}||^2 - ||P\mathbf{b}||^2

, which is answer D. Let's see why the other options fail: A

\mathbf{b}^T A \hat{\mathbf{x}} - ||A\hat{\mathbf{x}}||^2

incorrectly assumes you can factor the squared norm as a difference involving the inner product. B

||P\mathbf{b}||^2 - ||\mathbf{b}||^2

has the terms reversed and would give a negative value, which is impossible for a squared norm. C

||\mathbf{b}||^2 - \mathbf{b}^T A \hat{\mathbf{x}}

mixes a squared norm with an inner product without proper justification. Study tip: Always remember that in least squares problems,

\mathbf{b}

decomposes orthogonally into its projection onto the column space (

P\mathbf{b}

) plus the residual. Use the Pythagorean theorem to relate their squared norms.

Question 20

Consider the overdetermined system $A\mathbf{x} = \mathbf{b}$ where $A = \begin{pmatrix} 1 & 2 \\ 0 & 1 \\ 1 & 0 \end{pmatrix}$ and $\mathbf{b} = \begin{pmatrix} 3 \\ 1 \\ 2 \end{pmatrix}$ . If $\hat{\mathbf{x}}$ is the least squares solution obtained from the normal equations, what is the geometric interpretation of $A\hat{\mathbf{x}}$ ?

$A\hat{\mathbf{x}}$ is the orthogonal projection of $\mathbf{b}$ onto the column space of $A$ (correct answer)
$A\hat{\mathbf{x}}$ is the orthogonal projection of $\mathbf{b}$ onto the null space of $A^T$
$A\hat{\mathbf{x}}$ is the component of $\mathbf{b}$ that lies in the left null space of $A$
$A\hat{\mathbf{x}}$ is the reflection of $\mathbf{b}$ across the hyperplane defined by $A^T\mathbf{y} = \mathbf{0}$

Explanation: The least squares solution

\hat{\mathbf{x}}

minimizes

||A\mathbf{x} - \mathbf{b}||^2

, which means

A\hat{\mathbf{x}}

is the point in the column space of

A

closest to

\mathbf{b}

. This is precisely the orthogonal projection of

\mathbf{b}

onto

\text{Col}(A)

. Choice B is incorrect because the null space of

A^T

is orthogonal to the column space. Choice C is wrong because the left null space consists of vectors orthogonal to the row space. Choice D is incorrect because this describes a reflection, not a projection.