Statistics Graduate Level Quiz: Linear Regression In Matrix Form

9 questions · exam conditions

0:00

Linear Regression In Matrix FormQuestion 1 of 9

A full-column-rank design matrix $X$ is used to regress $y$ by ordinary least squares. Let $M_X=I-X(X^\top X)^{-1}X^\top$ , and suppose the current residual vector $r=M_Xy$ satisfies $r^\top r=20$ . A new predictor $z$ is added, where $z^\top M_Xz=5$ and $z^\top r=6$ . What is the residual sum of squares for the augmented model?

8

, obtained by subtracting

z^\top r+z^\top M_Xz

from the original sum of squares.

14

, obtained by subtracting the squared residual-predictor inner product divided by

z^\top r

\frac{64}{5}

, obtained by subtracting

\frac{(z^\top r)^2}{z^\top M_Xz}

from the original sum of squares.

\frac{76}{5}

, obtained by subtracting

\frac{z^\top M_Xz}{(z^\top r)^2}

after rescaling the predictor.

← Back to quizzes

Statistics Graduate Level Quiz

Statistics Graduate Level Quiz: Linear Regression In Matrix Form

Practice Linear Regression In Matrix Form in Statistics Graduate Level with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on Linear Regression In Matrix Form, giving you a quick way to practice the rules, question types, and explanations that matter most for Statistics Graduate Level.

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

$8$ , obtained by subtracting $z^\top r+z^\top M_Xz$ from the original sum of squares.
$14$ , obtained by subtracting the squared residual-predictor inner product divided by $z^\top r$ .
$\frac{64}{5}$ , obtained by subtracting $\frac{(z^\top r)^2}{z^\top M_Xz}$ from the original sum of squares. (correct answer)
$\frac{76}{5}$ , obtained by subtracting $\frac{z^\top M_Xz}{(z^\top r)^2}$ after rescaling the predictor.

Explanation: When you add a predictor to an OLS model, the new residual sum of squares drops by a precise, computable amount. The key formula comes from the Frisch-Waugh-Lovell theorem: the reduction in RSS when adding

z

to a model already containing

X

equals the squared correlation between

z

's residual (after projecting out

X

) and

y

's residual, scaled appropriately. Specifically, the new RSS is:

\text{RSS}_{\text{new}} = r^\top r - \frac{(z^\top r)^2}{z^\top M_X z}

Here,

M_X z

is the part of

z

orthogonal to the column space of

X

, and

z^\top r = z^\top M_X y

(since

r = M_X y

and

M_X

is idempotent). The reduction term is the OLS coefficient of

M_X y

M_X z

, squared and multiplied by

z^\top M_X z

— exactly one degree-of-freedom improvement. Plugging in:

20 - \frac{6^2}{5} = 20 - \frac{36}{5} = \frac{64}{5}

. This confirms C is correct. A is wrong because it subtracts

z^\top r + z^\top M_X z = 11

, which has no geometric or algebraic justification — it simply adds unrelated quantities. B inverts the correct formula, dividing by

z^\top r

instead of

z^\top M_X z

, conflating the numerator and denominator roles. D flips the fraction entirely, using

\frac{z^\top M_X z}{(z^\top r)^2}

, which is dimensionally inconsistent and shrinks RSS by a negligible amount. As a study strategy, memorize the added-variable reduction formula

\frac{(z^\top r)^2}{z^\top M_X z}

. On graduate exams, distractors are typically constructed by swapping numerator and denominator or combining terms additively — always check dimensional consistency.

Question 2

Consider ordinary least squares with hat matrix $H=X(X^\top X)^{-1}X^\top$ and residual-maker $M=I-H$ . The error covariance is $\sigma^2\Sigma$ , where $\Sigma$ is known, symmetric, and positive definite but need not equal the identity. Which condition is necessary and sufficient for the fitted vector $Hy$ and residual vector $My$ to be uncorrelated?

$\Sigma=cI$ for some positive constant $c$ , so the error covariance is a scalar multiple of the identity in every direction.
$H\Sigma=\Sigma H$ , so the covariance matrix commutes with the hat matrix, leaving the model space and its complement invariant under $\Sigma$ . (correct answer)
$X^\top\Sigma X$ is diagonal, so the predictors are mutually orthogonal under the covariance-weighted inner product defined by $\Sigma$ .
$H\Sigma H=\Sigma$ , so the entire error covariance structure is supported within the column space of the fitted-value projector.

Explanation: When checking whether two random vectors are uncorrelated, you compute their cross-covariance matrix and set it to zero. Here,

\text{Cov}(Hy, My) = H \cdot \text{Cov}(y,y) \cdot M^\top = \sigma^2 H\Sigma M

. Since

H

and

M = I - H

are symmetric and idempotent, this simplifies to

\sigma^2 H\Sigma(I-H) = \sigma^2(H\Sigma - H\Sigma H)

. For this to be the zero matrix, you need

H\Sigma = H\Sigma H

, which is equivalent to requiring

H\Sigma M = 0

. Notice that

H\Sigma = H\Sigma H

holds if and only if

H\Sigma(I-H) = 0

, and since

M = I - H

, this is precisely the condition

H\Sigma M = 0

. A clean way to see this: premultiplying

H\Sigma = H\Sigma H

by nothing and postmultiplying by

H

shows the condition is symmetric, ultimately equivalent to

H\Sigma = \Sigma H

— confirming answer B is both necessary and sufficient. Answer A is too restrictive. Proportionality to the identity guarantees commutativity, but commutativity holds in many cases where

\Sigma \neq cI

, so A is sufficient but not necessary. Answer C concerns the structure of

X^\top \Sigma X

, which governs generalized least squares efficiency, not whether OLS fitted values and residuals are uncorrelated. Diagonality of

X^\top \Sigma X

imposes no direct constraint on

H\Sigma M

. Answer D requires

H\Sigma H = \Sigma

, meaning

\Sigma

lives entirely within the column space of

X

. This is far too strong — and also not the right algebraic condition derived from the cross-covariance. A key study habit: always start uncorrelation problems by writing out the cross-covariance explicitly and simplifying using the idempotency and symmetry of projection matrices before evaluating any answer choice.

Question 3

Let a regression design be $X=[\mathbf 1,Z]$ , where $\mathbf 1$ is the intercept column. Define $P_1=\mathbf 1(\mathbf 1^\top\mathbf 1)^{-1}\mathbf 1^\top$ and $M_1=I-P_1$ . Assume $M_1Z$ has full column rank. Which expression is the orthogonal projector onto $\mathcal C(X)$ ?

$P_1+M_1Z(Z^\top M_1Z)^{-1}Z^\top M_1$ , which projects separately onto orthogonal intercept and centered-predictor spaces. (correct answer)
$P_1+Z(Z^\top Z)^{-1}Z^\top$ , which adds the intercept projector to the uncentered predictor projector.
$P_1-M_1Z(Z^\top M_1Z)^{-1}Z^\top M_1$ , which removes the centered-predictor component from the intercept fit.
$M_1+M_1Z(Z^\top M_1Z)^{-1}Z^\top M_1$ , which combines residual centering with centered-predictor projection.

Explanation: When you see a question about orthogonal projectors onto a partitioned column space, your instinct should be to apply the Frisch–Waugh–Lovell theorem and the formula for projecting onto

\mathcal{C}(X) = \mathcal{C}([\mathbf{1}, Z])

. The key idea is that you can build

P_X

by first projecting onto

\mathcal{C}(\mathbf{1})

, then projecting the residual of

Z

(after removing the intercept) onto its own column space. Formally,

M_1 Z

represents the columns of

Z

orthogonalized against

\mathbf{1}

. Since

\mathcal{C}(\mathbf{1})

and

\mathcal{C}(M_1 Z)

are orthogonal complements within

\mathcal{C}(X)

, the total projector decomposes as:

P_X = P_1 + P_{M_1 Z} = P_1 + M_1 Z(Z^\top M_1 Z)^{-1} Z^\top M_1

This is exactly answer A, which correctly identifies the two orthogonal subspaces and adds their projectors — a valid operation precisely because they are orthogonal. B is wrong because

P_1 + Z(Z^\top Z)^{-1}Z^\top

adds projectors onto non-orthogonal subspaces (

\mathbf{1} \in \mathcal{C}(Z)

in general), so the sum is not idempotent and does not define a valid orthogonal projector. C subtracts the centered-predictor component instead of adding it, which would remove directions from the fit rather than include them — this produces neither a projector nor the correct range. D replaces

P_1

with

M_1

, projecting onto the orthogonal complement of

\mathbf{1}

rather than onto

\mathbf{1}

itself, so the intercept subspace is entirely missing from the range. As a study tip: whenever the design matrix has a partitioned structure

[A, B]

, always orthogonalize

B

against

A

first — projectors add cleanly only when their ranges are orthogonal.

Question 4

Let $X$ be rank deficient, and let $G$ be any generalized inverse satisfying $(X^\top X)G(X^\top X)=X^\top X$ . Define $\tilde\beta=GX^\top y$ and $\tilde y=X\tilde\beta$ . Which statement holds for every such choice of $G$ ?

The coefficient vector $\tilde\beta$ is unique and has minimum Euclidean norm, although the fitted vector may depend on $G$ .
Both the fitted vector and coefficient vector are unique whenever $y$ lies in $\mathcal C(X)$ , but both may vary otherwise.
The fitted vector is unique only when $G$ is symmetric, whereas the coefficient vector is invariant across all generalized inverses.
The fitted vector $\tilde y$ is unique and equals the Euclidean projection of $y$ onto $\mathcal C(X)$ , although coefficients may vary. (correct answer)

Explanation: When working with rank-deficient design matrices, the central question is always: what quantities are estimable — meaning invariant across all valid generalized inverses? This question tests exactly that distinction. The key insight is that

\tilde{y} = X\tilde{\beta} = XGX^\top y

. Even though

G

is not unique, the product

XGX^\top

is unique for any generalized inverse of

X^\top X

. This is because

XGX^\top

is the orthogonal projection matrix onto

\mathcal{C}(X)

, which is entirely determined by the column space — not by any particular basis. Therefore,

\tilde{y}

is always the orthogonal projection of

y

onto

\mathcal{C}(X)

, regardless of which

G

you choose. This confirms D is correct. A is wrong on two counts:

\tilde{\beta} = GX^\top y

is generally not unique (it varies with

G

), and minimum-norm solutions require the Moore-Penrose pseudoinverse specifically — not just any generalized inverse. The fitted vector, not the coefficient vector, is the invariant quantity. B is incorrect because even when

y \in \mathcal{C}(X)

, the coefficients

\tilde{\beta}

can still vary across different generalized inverses (since the system

X\beta = y

has infinitely many solutions under rank deficiency). The fitted values are always unique, full stop — not conditionally. C has the invariance relationship backwards. It's the fitted values that are invariant, not the coefficients. Symmetry of

G

is irrelevant to this invariance. Your study anchor: in rank-deficient models, estimable functions of

\beta

— those of the form

X\beta

— are uniquely determined. Coefficients alone are not estimable, but fitted values always are.

Question 5

A reduced linear model has hat matrix $H_0$ and rank $3$ . A nested full model has hat matrix $H_1$ and rank $5$ , based on $n=25$ observations. Suppose $y^\top(H_1-H_0)y=12$ and the full-model residual sum of squares is $40$ . What is the partial $F$ statistic, including its numerator and denominator degrees of freedom?

$F=6.00$ with degrees of freedom $2$ and $20$ , using the unaveraged extra sum of squares.
$F=3.30$ with degrees of freedom $2$ and $22$ , using the reduced-model residual degrees of freedom.
$F=3.00$ with degrees of freedom $2$ and $20$ , using the rank difference and full-model error mean square. (correct answer)
$F=0.30$ with degrees of freedom $2$ and $20$ , using the ratio of total extra sum of squares to residual sum of squares.

Explanation: When comparing nested linear models, the partial

F

statistic measures whether the additional predictors in the full model explain significantly more variance. The formula is:

F = \frac{y^\top(H_1 - H_0)y / (r_1 - r_0)}{RSS_1 / (n - r_1)}

where

r_1 - r_0

is the rank difference (numerator df) and

n - r_1

is the full-model error df (denominator df). Here,

r_1 - r_0 = 5 - 3 = 2

, so the extra sum of squares must be averaged over 2 degrees of freedom:

12/2 = 6

. The full-model residual mean square is

RSS_1/(n - r_1) = 40/(25 - 5) = 40/20 = 2

. Therefore:

F = \frac{6}{2} = 3.00

with numerator df

= 2

and denominator df

= 20

. This confirms C is correct. Choice A fails because it places the unaveraged extra sum of squares (12) in the numerator rather than the mean square (6), inflating

F

to 6.00 — you must divide by the numerator degrees of freedom before forming the ratio. Choice B uses denominator df

= 22

, which corresponds to

n - r_0 = 25 - 3

, the reduced model's error df. The denominator must use the full model's error df

(n - r_1 = 20)

. Choice D takes the ratio of the total extra sum of squares to the total residual sum of squares without any averaging, which is not a valid

F

statistic construction at all. Study tip: Always track two separate averaging steps — divide the extra SS by

(r_1 - r_0)

and the residual SS by

(n - r_1)

— and confirm your denominator df comes from the full model, not the reduced one.

Question 6

A full-rank model containing design matrix $X$ is augmented by one predictor $z$ . Let $M_X=I-X(X^\top X)^{-1}X^\top$ . Suppose $z^\top M_Xz=4$ and $z^\top M_Xy=6$ . Which pair gives the coefficient of $z$ in the augmented regression and the corresponding reduction in residual sum of squares?

The coefficient is $\frac32$ , and the reduction in residual sum of squares is $9$ . (correct answer)
The coefficient is $\frac23$ , and the reduction in residual sum of squares is $4$ .
The coefficient is $\frac32$ , and the reduction in residual sum of squares is $6$ .
The coefficient is $\frac23$ , and the reduction in residual sum of squares is $9$ .

Explanation: When a model with design matrix

X

is augmented by a new predictor

z

, the Frisch-Waugh-Lovell (FWL) theorem tells you everything you need. The key insight is that the new coefficient and its contribution to fit depend entirely on the residualized versions of

z

and

y

— that is, what remains after projecting out

X

. The matrix

M_X = I - X(X^\top X)^{-1}X^\top

is exactly this residual-maker (the annihilator of

X

). By FWL, the OLS coefficient on

z

in the augmented model equals:

\hat{\gamma} = \frac{z^\top M_X y}{z^\top M_X z} = \frac{6}{4} = \frac{3}{2}

The reduction in residual sum of squares (RSS) from adding

z

— sometimes called the "extra sum of squares" — equals:

\Delta\text{RSS} = \hat{\gamma}^2 \cdot (z^\top M_X z) = \left(\frac{3}{2}\right)^2 \cdot 4 = \frac{9}{4} \cdot 4 = 9

Equivalently,

\Delta\text{RSS} = \frac{(z^\top M_X y)^2}{z^\top M_X z} = \frac{36}{4} = 9

. This confirms A is correct. Choice B inverts the ratio for

\hat{\gamma}

, giving

2/3

, and compounds the error with an incorrect RSS. Choice C gets the coefficient right but reports the RSS as 6 — confusing

z^\top M_X y

itself for the reduction in RSS. Choice D inverts the coefficient like B but accidentally gets the RSS right, making it a particularly tempting trap. Study tip: Memorize both FWL formulas together —

\hat{\gamma} = (z^\top M_X z)^{-1}z^\top M_X y

and

\Delta\text{RSS} = \hat{\gamma} \cdot z^\top M_X y

— since graduate exams routinely test whether you know both quantities and won't conflate them.

Question 7

In a full-rank ordinary least-squares fit, the residual sum of squares using all observations is $40$ . For observation $i$ , the ordinary residual is $e_i=2$ and the leverage is $h_{ii}=\frac14$ . The model is refitted after deleting observation $i$ . What is the residual sum of squares computed from the remaining observations?

$36$ , because deleting the observation removes its ordinary squared residual $e_i^2$ from the full-data sum.
$\frac{104}{3}$ , because deletion reduces the sum by $\frac{e_i^2}{1-h_{ii}}$ after accounting for refitting. (correct answer)
$\frac{116}{3}$ , because deletion reduces the sum by $e_i^2h_{ii}$ after leverage adjustment.
$\frac{296}{9}$ , because deletion reduces the sum by $\frac{e_i^2}{(1-h_{ii})^2}$ using the deleted residual.

Explanation: When you delete an observation from a regression model and refit, the new residual sum of squares is not simply the old RSS minus that observation's squared residual — the entire model changes. The key identity connecting full-data quantities to the deleted-model RSS is:

\text{RSS}_{(i)} = \text{RSS} - \frac{e_i^2}{1 - h_{ii}}

This formula works because the change in fit upon deletion depends not just on how large the residual is, but on how influential the observation is via its leverage

h_{ii}

. High-leverage points "pull" the fit toward themselves, so removing them causes a larger adjustment than their raw residual suggests. Plugging in

\text{RSS} = 40

e_i = 2

, and

h_{ii} = \frac{1}{4}

\frac{e_i^2}{1 - h_{ii}} = \frac{4}{1 - \frac{1}{4}} = \frac{4}{\frac{3}{4}} = \frac{16}{3}

\text{RSS}_{(i)} = 40 - \frac{16}{3} = \frac{120 - 16}{3} = \frac{104}{3}

This confirms answer B. Answer A makes the intuitive but incorrect assumption that deleting observation

i

simply removes

e_i^2 = 4

from the RSS, ignoring that refitting shifts all coefficient estimates. Answer C applies the wrong adjustment formula, subtracting

e_i^2 h_{ii} = 1

instead of the correct leverage-scaled term. Answer D uses

\frac{e_i^2}{(1-h_{ii})^2}

, which is the square of the internally studentized residual structure — a plausible-looking but incorrect expression for this quantity. A useful anchor: the deleted residual

e_{(i)} = \frac{e_i}{1-h_{ii}}

is the predicted residual for observation

i

from the model fit without it, and

\text{RSS}_{(i)} = \text{RSS} - e_i \cdot e_{(i)}

is the compact way to remember the correct formula.

Question 8

In a linear model with error covariance $\sigma^2\Sigma$ , where $\Sigma$ is known and positive definite, generalized least squares minimizes $(y-X\beta)^\top\Sigma^{-1}(y-X\beta)$ . If $\hat y=X\hat\beta_{\mathrm{GLS}}$ and $r=y-\hat y$ , which statement correctly describes the associated projection geometry in the original coordinates?

$X^\top r=0$ , and $\hat y$ is the ordinary Euclidean orthogonal projection of $y$ onto $\mathcal C(X)$ .
$X^\top\Sigma^{-2}r=0$ , and $\hat y$ is the Euclidean projection after weighting fitted values by $\Sigma^{-1}$ .
$X^\top\Sigma r=0$ , and $\hat y$ is the projection onto $\mathcal C(X)$ under the $\Sigma$ inner product.
$X^\top\Sigma^{-1}r=0$ , and $\hat y$ is the projection onto $\mathcal C(X)$ under the $\Sigma^{-1}$ inner product. (correct answer)

Explanation: Whenever you encounter a GLS problem involving projection geometry, your first instinct should be: which inner product defines "orthogonality" here? GLS minimizes a weighted quadratic form, so the geometry lives in a non-Euclidean space. The GLS estimator solves the normal equations derived by differentiating

(y - X\beta)^\top \Sigma^{-1}(y - X\beta)

with respect to

\beta

, yielding

X^\top \Sigma^{-1}(y - X\hat\beta_{\mathrm{GLS}}) = 0

, or equivalently

X^\top \Sigma^{-1} r = 0

. This is the orthogonality condition under the inner product

\langle u, v \rangle_{\Sigma^{-1}} = u^\top \Sigma^{-1} v

. Under this inner product,

\hat y

is the projection of

y

onto

\mathcal{C}(X)

— meaning the residual

r

is "perpendicular" to the column space of

X

in the

\Sigma^{-1}

-metric sense. This confirms D is correct. Choice A describes ordinary least squares (OLS), where

\Sigma = I

and orthogonality is Euclidean. It incorrectly applies standard projection to a setting that requires a weighted metric. Choice B introduces

\Sigma^{-2}

, which appears nowhere in the GLS derivation — this is a fabricated condition with no geometric or algebraic basis. Choice C flips the matrix: using

\Sigma

instead of

\Sigma^{-1}

would correspond to minimizing

(y - X\beta)^\top \Sigma(y - X\beta)

, a different problem entirely — and it incorrectly names the resulting inner product. A reliable study tip: always trace the normal equations directly from the objective function. The matrix appearing in the weight of the quadratic form is exactly the matrix that appears in the orthogonality condition

X^\top W r = 0

. Here,

W = \Sigma^{-1}

, so the answer follows immediately.

Question 9

For a full-column-rank linear model, let $\hat\beta$ be the unrestricted least-squares estimator. A restricted fit is obtained by imposing $a^\top\beta=d$ . Suppose $a^\top\hat\beta-d=3$ and $a^\top(X^\top X)^{-1}a=2$ . By how much does the residual sum of squares increase under the restriction?

$\frac32$ , obtained by dividing the constraint discrepancy by its quadratic-form scaling factor.
$3$ , obtained by using the unsquared distance from the unrestricted estimate to the constraint.
$\frac92$ , obtained from the squared discrepancy divided by the quadratic-form scaling factor. (correct answer)
$18$ , obtained by multiplying the squared discrepancy by the quadratic-form scaling factor.

Explanation: When you impose a linear restriction on a least-squares problem, the restricted estimator must "bend" away from the unrestricted optimum, and that detour always increases the residual sum of squares. The exact formula for this increase is derived from the Lagrangian approach to constrained least squares, and it's worth memorizing: the RSS increases by

\Delta RSS = \frac{(a^\top\hat\beta - d)^2}{a^\top(X^\top X)^{-1}a}.

This expression has an intuitive structure — the numerator measures how far the unrestricted estimate violates the constraint, squared, and the denominator is a scaling factor that accounts for the geometry of the design matrix along the direction

a

. Plugging in the given values, you get

\frac{3^2}{2} = \frac{9}{2}

, confirming that C is correct. Looking at the wrong choices: A divides the unsquared discrepancy (3) by the scaling factor (2), giving

\frac{3}{2}

. This forgets to square the numerator — a classic algebra slip that loses the squared-error structure of least squares. B reports the raw discrepancy of 3 with no division at all, ignoring both the squaring and the scaling factor entirely — this would only matter if the formula were something far simpler. D multiplies the squared discrepancy by the scaling factor, getting

9 \times 2 = 18

, which inverts the correct relationship; larger

a^\top(X^\top X)^{-1}a

actually reduces the penalty, not amplifies it. A reliable study tip: whenever you see a restricted vs. unrestricted RSS comparison, immediately write down

\Delta RSS = (a^\top\hat\beta - d)^2 / [a^\top(X^\top X)^{-1}a]

and match it to an F-statistic numerator — that connection will help you remember both the formula and its role in hypothesis testing.