Statistics Graduate Level Quiz: Least Squares And Normal Equations

10 questions · exam conditions

0:00

Least Squares And Normal EquationsQuestion 1 of 10

An existing full-rank least squares fit has $\widehat{\beta}=(1,2)^{\mathsf T}$ and $(X^{\mathsf T}X)^{-1}=I_2$ . One new observation is added with predictor row $x^{\mathsf T}=(1,1)$ and response $y=5$ . What is the updated least squares estimate?

(5/3,8/3)^{\mathsf T}

(3,4)^{\mathsf T}

(2,3)^{\mathsf T}

(4/3,7/3)^{\mathsf T}

← Back to quizzes

Statistics Graduate Level Quiz

Statistics Graduate Level Quiz: Least Squares And Normal Equations

Practice Least Squares And Normal Equations in Statistics Graduate Level with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on Least Squares And Normal Equations, giving you a quick way to practice the rules, question types, and explanations that matter most for Statistics Graduate Level.

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

$(5/3,8/3)^{\mathsf T}$ (correct answer)
$(3,4)^{\mathsf T}$
$(2,3)^{\mathsf T}$
$(4/3,7/3)^{\mathsf T}$

Explanation: When a single new observation is added to an existing least squares fit, you can avoid refitting from scratch using the rank-1 update formula. Given current estimate

\widehat{\beta}_{\text{old}}

, matrix

(X^\mathsf{T}X)^{-1}

, new predictor

x

, and new response

y

, the updated inverse is:

M_{\text{new}} = \left(X^\mathsf{T}X + xx^\mathsf{T}\right)^{-1}

and the updated estimate is:

\widehat{\beta}_{\text{new}} = M_{\text{new}}\left(X^\mathsf{T}X\,\widehat{\beta}_{\text{old}} + xy\right)

Here,

X^\mathsf{T}X = I_2

, so

X^\mathsf{T}X\,\widehat{\beta}_{\text{old}} = (1,2)^\mathsf{T}

. Adding

xy = (1,1)^\mathsf{T}\cdot 5 = (5,5)^\mathsf{T}

gives the updated sufficient statistic

(6,7)^\mathsf{T}

. Next,

X^\mathsf{T}X + xx^\mathsf{T} = I_2 + \begin{pmatrix}1\\1\end{pmatrix}(1,1) = \begin{pmatrix}2&1\\1&2\end{pmatrix}

, whose inverse is $$\frac{1}{3}\begin{pmatrix}2&-1\-1&2\end{pmatrix}

$$\widehat{\beta}_{\text{new}} = \frac{1}{3}\begin{pmatrix}2&-1\\-1&2\end{pmatrix}\begin{pmatrix}6\\7\end{pmatrix} = \frac{1}{3}\begin{pmatrix}5\\8\end{pmatrix} = \left(\frac{5}{3},\frac{8}{3}\right)^\mathsf{T}$$ This confirms **answer A**. Choice B $$(3,4)^\mathsf{T}$$ results from forgetting to apply $$M_{\text{new}}$$ and simply adding $$xy$$ to $$\widehat{\beta}_{\text{old}}$$. Choice C $$(2,3)^\mathsf{T}$$ comes from naively averaging the old estimate with the new observation's implied coefficients. Choice D $$(4/3, 7/3)^\mathsf{T}$$ arises from a sign error when inverting the updated matrix. **Study tip:** Memorize the rank-1 (Sherman-Morrison) update structure — on graduate exams, sequential updating questions almost always hinge on correctly computing $$(X^\mathsf{T}X + xx^\mathsf{T})^{-1}$$ and updating both the information matrix *and* the sufficient statistic simultaneously.

Question 2

A simple linear regression with intercept is fit using predictor $x$ . The sufficient statistics are $n=5$ , $\sum x_i=10$ , $\sum x_i^2=30$ , $\sum y_i=20$ , and $\sum x_i y_i=50$ . The same fitted values are to be expressed using the centered predictor $z_i=x_i-2$ in the model $y_i=\gamma_0+\gamma_1 z_i+e_i$ . What are the least squares estimates?

$(\widehat{\gamma}_0,\widehat{\gamma}_1)=(2,1)$
$(\widehat{\gamma}_0,\widehat{\gamma}_1)=(4,1)$ (correct answer)
$(\widehat{\gamma}_0,\widehat{\gamma}_1)=(0,1)$
$(\widehat{\gamma}_0,\widehat{\gamma}_1)=(4,-1)$

Explanation: When you reparametrize a regression by centering the predictor, the slope is invariant but the intercept shifts to reflect the new origin. This is the key insight: fitting

y_i = \gamma_0 + \gamma_1 z_i + e_i

with

z_i = x_i - \bar{x}

produces the same fitted values as the original model, so

\widehat{\gamma}_1 = \widehat{\beta}_1

, and

\widehat{\gamma}_0 = \bar{y}

(since the regression line always passes through

(\bar{z}, \bar{y}) = (0, \bar{y})

). First, compute the sufficient statistics you need. From the given data:

\bar{x} = 10/5 = 2

\bar{y} = 20/5 = 4

. The slope from the original model is:

\widehat{\beta}_1 = \frac{\sum x_i y_i - n\bar{x}\bar{y}}{\sum x_i^2 - n\bar{x}^2} = \frac{50 - 5(2)(4)}{30 - 5(4)} = \frac{10}{10} = 1

Since

z_i = x_i - 2 = x_i - \bar{x}

, the centered predictor has mean zero, so

\widehat{\gamma}_1 = \widehat{\beta}_1 = 1

and

\widehat{\gamma}_0 = \bar{y} = 4

. The answer is (B). Choice (A) sets

\widehat{\gamma}_0 = 2 = \bar{x}

, confusing the mean of

x

with the mean of

y

. Choice (C) sets the intercept to zero, which would only hold if

\bar{y} = 0

. Choice (D) incorrectly negates the slope, perhaps from a sign error in the centering formula. Your takeaway: centering a predictor never changes the slope — it only re-anchors the intercept to

\bar{y}

. Memorize that

\widehat{\gamma}_0 = \bar{y}

when you center at

\bar{x}

; it's a frequent exam shortcut.

Question 3

A through-the-origin model $y_i=\beta x_i+e_i$ is fit by minimizing $\sum_i w_i(y_i-\beta x_i)^2$ . Two observations have $(x_1,y_1,w_1)=(1,1,4)$ and $(x_2,y_2,w_2)=(2,4,1)$ . What is the weighted least squares estimate of $\beta$ ?

$6/5$
$33/17$
$9/5$
$3/2$ (correct answer)

Explanation: Weighted least squares (WLS) generalizes ordinary least squares by giving each observation a weight that reflects its precision or reliability. When you see a WLS problem, your first instinct should be to write down the closed-form estimator rather than re-deriving it from scratch. For a through-the-origin model, minimizing

\sum_i w_i(y_i - \beta x_i)^2

with respect to

\beta

yields the estimator:

\hat{\beta} = \frac{\sum_i w_i x_i y_i}{\sum_i w_i x_i^2}

This is simply the OLS formula with each term weighted by

w_i

. Plugging in the two observations:

\sum_i w_i x_i y_i = (4)(1)(1) + (1)(2)(4) = 4 + 8 = 12

\sum_i w_i x_i^2 = (4)(1)^2 + (1)(2)^2 = 4 + 4 = 8

\hat{\beta} = \frac{12}{8} = \frac{3}{2}

This confirms D is correct. Now for the distractors. A (

6/5

) likely results from mistakenly using

\sum w_i y_i / \sum w_i x_i

, forgetting to square the

x_i

terms in the denominator. C (

9/5

) comes from ignoring weights entirely and computing the unweighted OLS estimate

\sum x_i y_i / \sum x_i^2 = (1\cdot1 + 2\cdot4)/(1+4) = 9/5

. B (

33/17

) appears to arise from a scrambled weighting scheme, perhaps incorrectly applying weights to only the numerator or denominator but not both consistently. A reliable study tip: always write the WLS normal equations first. The formula

\hat{\beta} = (\mathbf{X}^T \mathbf{W} \mathbf{X})^{-1} \mathbf{X}^T \mathbf{W} \mathbf{y}

reduces to this ratio in the simple no-intercept case — memorizing this structure prevents most computational errors.

Question 4

For a full-rank least squares problem, suppose the unconstrained estimate is $\widehat{\beta}=(2,1)^{\mathsf T}$ and $X^{\mathsf T}X=\operatorname{diag}(1,4)$ . The model is refit subject to the restriction $\beta_1+\beta_2=0$ . What is the restricted least squares estimate?

$(1/2,-1/2)^{\mathsf T}$
$(2,-2)^{\mathsf T}$
$(-2/5,2/5)^{\mathsf T}$ (correct answer)
$(-1,1)^{\mathsf T}$

Explanation: When you encounter a restricted least squares problem, your goal is to minimize the usual sum of squares subject to a linear constraint. For a constraint of the form

R\beta = r

, the restricted estimator has the closed-form solution:

\tilde{\beta} = \widehat{\beta} - (X^\mathsf{T}X)^{-1}R^\mathsf{T}\left[R(X^\mathsf{T}X)^{-1}R^\mathsf{T}\right]^{-1}(R\widehat{\beta} - r)

Here, the constraint

\beta_1 + \beta_2 = 0

means

R = (1,1)

and

r = 0

. With

X^\mathsf{T}X = \text{diag}(1,4)

, its inverse is

\text{diag}(1, 1/4)

. First compute

R\widehat{\beta} - r = 2 + 1 - 0 = 3

. Then

R(X^\mathsf{T}X)^{-1}R^\mathsf{T} = 1(1) + 1(1/4) = 5/4

, so the scalar inverse is

4/5

. The correction term becomes

(X^\mathsf{T}X)^{-1}R^\mathsf{T} \cdot (4/5) \cdot 3 = \text{diag}(1,1/4)(1,1)^\mathsf{T} \cdot (12/5) = (12/5, 3/5)^\mathsf{T}

. Subtracting:

\tilde{\beta} = (2,1)^\mathsf{T} - (12/5, 3/5)^\mathsf{T} = (-2/5, 2/5)^\mathsf{T}

, confirming C. Choice A,

(1/2, -1/2)^\mathsf{T}

, satisfies the constraint but ignores the differential weighting from

X^\mathsf{T}X

— it's what you'd get assuming an identity matrix. Choice B,

(2,-2)^\mathsf{T}

, incorrectly anchors

\beta_1

at its unconstrained value and simply negates it. Choice D,

(-1,1)^\mathsf{T}

, satisfies the constraint but corresponds to no principled derivation given the structure of

X^\mathsf{T}X

. As a strategy, always remember that

X^\mathsf{T}X

acts as a precision matrix — parameters estimated more precisely (larger diagonal entries) get pulled less by the constraint. Here,

\beta_2

has four times the precision, so it shifts less, which is exactly what you see in answer C.

Question 5

A regression of $y$ on an intercept, $x$ , and $z$ is to be fit by ordinary least squares. After centering all three variables, the cross-products are $x^{\mathsf T}x=10$ , $z^{\mathsf T}z=5$ , $x^{\mathsf T}z=4$ , $x^{\mathsf T}y=8$ , and $z^{\mathsf T}y=3$ .

What is the least squares coefficient of $x$ in the multiple regression?

$4/5$
$8/5$
$5/7$
$14/17$ (correct answer)

Explanation: When you see a multiple regression problem with centered variables and cross-products given, your instinct should be to set up the normal equations directly. Centering eliminates the intercept from the system, so you only need to solve for the slope coefficients. The normal equations for centered variables are:

\begin{pmatrix} x^\mathsf{T}x & x^\mathsf{T}z \\ x^\mathsf{T}z & z^\mathsf{T}z \end{pmatrix} \begin{pmatrix} \hat{\beta}_x \\ \hat{\beta}_z \end{pmatrix} = \begin{pmatrix} x^\mathsf{T}y \\ z^\mathsf{T}y \end{pmatrix}

Substituting the given values:

\begin{pmatrix} 10 & 4 \\ 4 & 5 \end{pmatrix} \begin{pmatrix} \hat{\beta}_x \\ \hat{\beta}_z \end{pmatrix} = \begin{pmatrix} 8 \\ 3 \end{pmatrix}

The determinant is

10(5) - 4(4) = 50 - 16 = 34

. Using Cramer's rule,

\hat{\beta}_x = \frac{8(5) - 3(4)}{34} = \frac{40 - 12}{34} = \frac{28}{34} = \frac{14}{17}

, confirming answer D. Now for the distractors. Choice A (

4/5

) comes from naively dividing

x^\mathsf{T}y

z^\mathsf{T}z

— mixing up which cross-product matrix entry belongs where. Choice B (

8/5

) is the simple regression coefficient of

y

x

alone (

x^\mathsf{T}y / x^\mathsf{T}x = 8/10

... actually it equals

8/10

), or more likely from ignoring the off-diagonal terms entirely. Choice C (

5/7

) likely arises from an arithmetic error in computing the determinant or numerator, perhaps using

35/49

from a sign mistake. The key study tip: in multiple regression, you cannot simply divide

x^\mathsf{T}y

x^\mathsf{T}x

as you would in simple regression. The off-diagonal terms in

(X^\mathsf{T}X)^{-1}

adjust for the correlation between predictors — always solve the full system.

Question 6

A regression model contains an intercept and a treatment indicator $D_i$ , where $D_i=1$ for treated observations and $D_i=0$ for control observations. Additional predictors may also be present. The model is fit by ordinary least squares.

Which statement about the least squares residuals must hold, regardless of the additional predictors?

Only the residual sum over all observations must equal zero.
The treated and control residual sums must each equal zero. (correct answer)
The two group residual sums must be equal but need not be zero.
Only the treated-group residual sum must equal zero.

Explanation: Whenever a regression model includes an intercept and a binary group indicator

D_i

, you should immediately think about the normal equations — the system of equations OLS solves to minimize the sum of squared residuals. Each predictor in the model, including the intercept and

D_i

itself, generates its own normal equation by requiring that the predictor is orthogonal to the residuals. The intercept term produces the equation

\sum_i \hat{e}_i = 0

, guaranteeing the overall residual sum is zero. But

D_i

generates its own separate normal equation:

\sum_i D_i \hat{e}_i = 0

, which means the sum of residuals among treated units only equals zero. Combining these two — since

\sum_i \hat{e}_i = \sum_{D_i=0} \hat{e}_i + \sum_{D_i=1} \hat{e}_i = 0

and

\sum_{D_i=1} \hat{e}_i = 0

— you can immediately deduce that

\sum_{D_i=0} \hat{e}_i = 0

as well. So both group residual sums must individually equal zero, confirming answer B. Answer A is incomplete — while the overall sum is zero, A misses the stronger constraint that each group sum is separately zero. Answer C is doubly wrong: the two group sums are not merely equal to each other but are each independently zero. Answer D is tempting because the normal equation for

D_i

directly pins down the treated sum, but it neglects that the control sum is then forced to zero by the intercept equation. Study tip: Any time a binary group indicator is included in an OLS model with an intercept, remember that OLS perfectly fits the group means — a powerful fact that implies both group residuals sum to zero, regardless of other covariates.

Question 7

A linear model includes a known offset: $y_i=o_i+\beta_0+\beta_1x_i+e_i$ . For four observations, the summaries are $\sum x_i=2$ , $\sum x_i^2=6$ , $\sum y_i=10$ , $\sum x_i y_i=9$ , $\sum o_i=2$ , and $\sum x_i o_i=1$ .

What are the ordinary least squares estimates of $\beta_0$ and $\beta_1$ ?

$(\widehat{\beta}_0,\widehat{\beta}_1)=(8/5,\;4/5)$ (correct answer)
$(\widehat{\beta}_0,\widehat{\beta}_1)=(21/10,\;4/5)$
$(\widehat{\beta}_0,\widehat{\beta}_1)=(13/5,\;4/5)$
$(\widehat{\beta}_0,\widehat{\beta}_1)=(0,\;4/3)$

Explanation: When a model includes a known offset

o_i

, OLS treats it as a fixed, non-estimated component. The trick is to absorb the offset into the response by defining

y_i^* = y_i - o_i

, then regress

y_i^*

x_i

with an intercept. Your adjusted summaries become:

\sum y_i^* = \sum y_i - \sum o_i = 10 - 2 = 8

and

\sum x_i y_i^* = \sum x_i y_i - \sum x_i o_i = 9 - 1 = 8

, with

n=4

\sum x_i = 2

\sum x_i^2 = 6

. The standard OLS normal equations give:

\widehat{\beta}_1 = \frac{n\sum x_i y_i^* - \sum x_i \sum y_i^*}{n\sum x_i^2 - (\sum x_i)^2} = \frac{4(8) - (2)(8)}{4(6) - 4} = \frac{16}{20} = \frac{4}{5}

\widehat{\beta}_0 = \frac{\sum y_i^* - \widehat{\beta}_1 \sum x_i}{n} = \frac{8 - \frac{4}{5}(2)}{4} = \frac{8 - 8/5}{4} = \frac{32/5}{4} = \frac{8}{5}

This confirms answer A:

(\widehat{\beta}_0, \widehat{\beta}_1) = (8/5,\, 4/5)

. Choice B likely arises from an arithmetic error when computing

\widehat{\beta}_0

, perhaps subtracting

\sum o_i

incorrectly from only part of the expression. Choice C comes from forgetting to subtract the offset at all — using raw

\sum y_i

and

\sum x_i y_i

directly. Choice D results from an entirely different misapplication, forcing the intercept to zero or misidentifying which parameters to estimate. Your go-to strategy: whenever you see an offset term, immediately redefine

y^* = y - o

and run standard OLS on

y^*

. Don't let the notation obscure what is fundamentally a simple linear regression after that substitution.

Question 8

A least squares model with design matrix $X$ has residual vector $e$ and residual sum of squares $12$ . A candidate predictor $z$ is added. Let $P_X$ denote the orthogonal projection onto the column space of $X$ . It is known that $e^{\mathsf T}z=2$ , $z^{\mathsf T}z=10$ , and $z^{\mathsf T}P_Xz=6$ .

What is the residual sum of squares after adding $z$ to the model?

$8$
$10$
$11$ (correct answer)
$58/5$

Explanation: When you add a new predictor

z

to a least squares model, the new RSS equals the old RSS minus the reduction in RSS attributable to

z

's component orthogonal to the existing column space. The key formula is:

\text{RSS}_{\text{new}} = \text{RSS}_{\text{old}} - \frac{(e^\mathsf{T}z)^2}{z^\mathsf{T}(I - P_X)z}

Here's why: the residual vector

e = (I - P_X)y

is already orthogonal to

\text{col}(X)

, so only the part of

z

outside

\text{col}(X)

—namely

(I - P_X)z

—can further reduce the RSS. The reduction equals the squared correlation between

e

and

(I-P_X)z

, normalized by the length of

(I-P_X)z

. Computing the denominator:

z^\mathsf{T}(I - P_X)z = z^\mathsf{T}z - z^\mathsf{T}P_Xz = 10 - 6 = 4

. The reduction is therefore

\frac{(e^\mathsf{T}z)^2}{z^\mathsf{T}(I-P_X)z} = \frac{2^2}{4} = \frac{4}{4} = 1

. So

\text{RSS}_{\text{new}} = 12 - 1 = 11

, confirming C is correct. Choice A (8) results from incorrectly using

z^\mathsf{T}z = 10

as the denominator, ignoring the projection. Choice B (10) likely comes from subtracting

e^\mathsf{T}z = 2

directly from the RSS, which has no geometric justification. Choice D (58/5) may arise from computing

12 - 4/10

, mistakenly using

z^\mathsf{T}P_Xz

in the denominator instead of

z^\mathsf{T}(I-P_X)z

. Always remember: it's the residualized portion of

z

—not

z

itself—that drives the reduction in RSS.

Question 9

In a rank-deficient least squares problem, the columns of the design matrix satisfy $X_3=X_1+X_2$ . One solution of the normal equations is $b_0=(1,1,1)^{\mathsf T}$ . Which coefficient vector is the minimum-Euclidean-norm least squares solution?

$(1,1,1)^{\mathsf T}$
$(2/3,2/3,4/3)^{\mathsf T}$ (correct answer)
$(4/3,4/3,2/3)^{\mathsf T}$
$(0,0,2)^{\mathsf T}$

Explanation: When a design matrix is rank-deficient, the normal equations have infinitely many solutions. The minimum-norm least squares solution is the unique solution with smallest Euclidean norm, given by the Moore-Penrose pseudoinverse:

b^+ = X^+y

. Critically, it lies in the row space of

X

— orthogonal to the null space. Here, the constraint

X_3 = X_1 + X_2

means the null space of

X

contains the vector

n = (1, 1, -1)^{\mathsf T}

(scaled appropriately). Specifically, any vector of the form

(1,1,-1)^{\mathsf T}t

added to a solution still satisfies the normal equations. The general solution is

b_0 + t(1,1,-1)^{\mathsf T} = (1+t,\, 1+t,\, 1-t)^{\mathsf T}

. To find the minimum-norm solution, minimize

\|b\|^2 = (1+t)^2 + (1+t)^2 + (1-t)^2

over

t

. Taking the derivative and setting it to zero:

2(1+t) + 2(1+t) + 2(1-t)(-1) = 0

, which gives

4(1+t) - 2(1-t) = 0 \Rightarrow 4+4t-2+2t = 0 \Rightarrow t = -1/3

. Substituting:

(1-1/3,\, 1-1/3,\, 1+1/3)^{\mathsf T} = (2/3,\, 2/3,\, 4/3)^{\mathsf T}

. That's answer B. A is wrong because

(1,1,1)^{\mathsf T}

is merely one particular solution — it has no special minimality property. C

(4/3,4/3,2/3)^{\mathsf T}

corresponds to

t = +1/3

, the wrong sign. D

(0,0,2)^{\mathsf T}

uses

t = -1

, which doesn't minimize the norm. Study tip: On rank-deficient problems, always parameterize the full solution family using the null space vector, then minimize the squared norm — this one calculus step separates the minimum-norm solution from all others.

Question 10

A full-rank ordinary least squares model has response vector $y$ , design matrix $X$ , coefficient estimate $\widehat{\beta}$ , and residual vector $e$ . The response is replaced by $y^*=y+Xc+u$ , where $c$ is fixed and $X^{\mathsf T}u=0$ .

Which pair gives the new coefficient estimate and new residual vector?

$(\widehat{\beta}+c,\ e+u)$ (correct answer)
$(\widehat{\beta}+c,\ e)$
$(\widehat{\beta},\ e+Xc+u)$
$(\widehat{\beta}-c,\ e+u+2Xc)$

Explanation: When you modify the response vector in OLS, you need to track how the projection machinery responds to each component separately. The key tool is decomposing

y^* = y + Xc + u

and applying the hat matrix

H = X(X^\mathsf{T}X)^{-1}X^\mathsf{T}

. The new coefficient estimate is

\widehat{\beta}^* = (X^\mathsf{T}X)^{-1}X^\mathsf{T}y^*

. Expanding:

(X^\mathsf{T}X)^{-1}X^\mathsf{T}(y + Xc + u) = \widehat{\beta} + c + (X^\mathsf{T}X)^{-1}X^\mathsf{T}u

. Since

X^\mathsf{T}u = 0

, the last term vanishes, giving

\widehat{\beta}^* = \widehat{\beta} + c

. For the residuals,

e^* = y^* - X\widehat{\beta}^* = (y + Xc + u) - X(\widehat{\beta} + c) = y - X\widehat{\beta} + u = e + u

. So the answer is A: the new estimate is

\widehat{\beta} + c

and the new residual is

e + u

. B gets the coefficient right but claims the residual is just

e

, ignoring that

u

is orthogonal to the column space of

X

— which means it lives entirely in the residual space and cannot be absorbed by the fitted values. C leaves

\widehat{\beta}

unchanged, forgetting that

Xc

is in the column space of

X

and is fully absorbed into the coefficient estimate, not the residual. D subtracts

c

rather than adding it, a sign error that reflects confusion about the direction of the projection update. A useful pattern to internalize: any signal in the column space of

X

shifts the coefficients; any signal orthogonal to

X

passes entirely into the residuals. Keeping this decomposition sharp will serve you across many OLS perturbation problems.