Linear Regression in Matrix Form - Statistics Graduate Level

1. Let $X$ be rank deficient, and let $G$ be any generalized inverse satisfying $(X^\top X)G(X^\top X)=X^\top X$. Define $\tilde\beta=GX^\top y$ and $\tilde y=X\tilde\beta$. Which statement holds for every such choice of $G$?
   Answer: The fitted vector $\tilde y$ is unique and equals the Euclidean projection of $y$ onto $\mathcal C(X)$, although coefficients may vary. When working with rank-deficient design matrices, the central question is always: what quantities are estimable — meaning invariant across all valid generalized inverses? This question tests exactly that distinction. The key insight is that $\tilde{y} = X\tilde{\beta} = XGX^\top y$. Even though $G$ is not unique, the product $XGX^\top$ is unique for any generalized inverse of $X^\top X$. This is because $XGX^\top$ is the orthogonal projection matrix onto $\mathcal{C}(X)$, which is entirely determined by the column space — not by any particular basis. Therefore, $\tilde{y}$ is always the orthogonal projection of $y$ onto $\mathcal{C}(X)$, regardless of which $G$ you choose. This confirms D is correct. A is wrong on two counts: $\tilde{\beta} = GX^\top y$ is generally not unique (it varies with $G$), and minimum-norm solutions require the Moore-Penrose pseudoinverse specifically — not just any generalized inverse. The fitted vector, not the coefficient vector, is the invariant quantity. B is incorrect because even when $y \in \mathcal{C}(X)$, the coefficients $\tilde{\beta}$ can still vary across different generalized inverses (since the system $X\beta = y$ has infinitely many solutions under rank deficiency). The fitted values are always unique, full stop — not conditionally. C has the invariance relationship backwards. It's the fitted values that are invariant, not the coefficients. Symmetry of $G$ is irrelevant to this invariance. Your study anchor: in rank-deficient models, estimable functions of $\beta$ — those of the form $X\beta$ — are uniquely determined. Coefficients alone are not estimable, but fitted values always are.
2. A full-column-rank design matrix $X$ is used to regress $y$ by ordinary least squares. Let $M_X=I-X(X^\top X)^{-1}X^\top$, and suppose the current residual vector $r=M_Xy$ satisfies $r^\top r=20$. A new predictor $z$ is added, where $z^\top M_Xz=5$ and $z^\top r=6$. What is the residual sum of squares for the augmented model?
   Answer: $\frac{64}{5}$, obtained by subtracting $\frac{(z^\top r)^2}{z^\top M_Xz}$ from the original sum of squares. When you add a predictor to an OLS model, the new residual sum of squares drops by a precise, computable amount. The key formula comes from the Frisch-Waugh-Lovell theorem: the reduction in RSS when adding $z$ to a model already containing $X$ equals the squared correlation between $z$'s residual (after projecting out $X$) and $y$'s residual, scaled appropriately. Specifically, the new RSS is: $\text{RSS}_{\text{new}} = r^\top r - \frac{(z^\top r)^2}{z^\top M_X z}$ Here, $M_X z$ is the part of $z$ orthogonal to the column space of $X$, and $z^\top r = z^\top M_X y$ (since $r = M_X y$ and $M_X$ is idempotent). The reduction term is the OLS coefficient of $M_X y$ on $M_X z$, squared and multiplied by $z^\top M_X z$ — exactly one degree-of-freedom improvement. Plugging in: $20 - \frac{6^2}{5} = 20 - \frac{36}{5} = \frac{64}{5}$. This confirms C is correct. A is wrong because it subtracts $z^\top r + z^\top M_X z = 11$, which has no geometric or algebraic justification — it simply adds unrelated quantities. B inverts the correct formula, dividing by $z^\top r$ instead of $z^\top M_X z$, conflating the numerator and denominator roles. D flips the fraction entirely, using $\frac{z^\top M_X z}{(z^\top r)^2}$, which is dimensionally inconsistent and shrinks RSS by a negligible amount. As a study strategy, memorize the added-variable reduction formula $\frac{(z^\top r)^2}{z^\top M_X z}$. On graduate exams, distractors are typically constructed by swapping numerator and denominator or combining terms additively — always check dimensional consistency.
3. For a full-column-rank linear model, let $\hat\beta$ be the unrestricted least-squares estimator. A restricted fit is obtained by imposing $a^\top\beta=d$. Suppose $a^\top\hat\beta-d=3$ and $a^\top(X^\top X)^{-1}a=2$. By how much does the residual sum of squares increase under the restriction?
   Answer: $\frac92$, obtained from the squared discrepancy divided by the quadratic-form scaling factor. When you impose a linear restriction on a least-squares problem, the restricted estimator must "bend" away from the unrestricted optimum, and that detour always increases the residual sum of squares. The exact formula for this increase is derived from the Lagrangian approach to constrained least squares, and it's worth memorizing: the RSS increases by $\Delta RSS = \frac{(a^\top\hat\beta - d)^2}{a^\top(X^\top X)^{-1}a}.$ This expression has an intuitive structure — the numerator measures how far the unrestricted estimate violates the constraint, squared, and the denominator is a scaling factor that accounts for the geometry of the design matrix along the direction $a$. Plugging in the given values, you get $\frac{3^2}{2} = \frac{9}{2}$, confirming that C is correct. Looking at the wrong choices: A divides the unsquared discrepancy (3) by the scaling factor (2), giving $\frac{3}{2}$. This forgets to square the numerator — a classic algebra slip that loses the squared-error structure of least squares. B reports the raw discrepancy of 3 with no division at all, ignoring both the squaring and the scaling factor entirely — this would only matter if the formula were something far simpler. D multiplies the squared discrepancy by the scaling factor, getting $9 \times 2 = 18$, which inverts the correct relationship; larger $a^\top(X^\top X)^{-1}a$ actually reduces the penalty, not amplifies it. A reliable study tip: whenever you see a restricted vs. unrestricted RSS comparison, immediately write down $\Delta RSS = (a^\top\hat\beta - d)^2 / [a^\top(X^\top X)^{-1}a]$ and match it to an F-statistic numerator — that connection will help you remember both the formula and its role in hypothesis testing.
4. In a linear model with error covariance $\sigma^2\Sigma$, where $\Sigma$ is known and positive definite, generalized least squares minimizes $(y-X\beta)^\top\Sigma^{-1}(y-X\beta)$. If $\hat y=X\hat\beta_{\mathrm{GLS}}$ and $r=y-\hat y$, which statement correctly describes the associated projection geometry in the original coordinates?
   Answer: $X^\top\Sigma^{-1}r=0$, and $\hat y$ is the projection onto $\mathcal C(X)$ under the $\Sigma^{-1}$ inner product. Whenever you encounter a GLS problem involving projection geometry, your first instinct should be: which inner product defines "orthogonality" here? GLS minimizes a weighted quadratic form, so the geometry lives in a non-Euclidean space. The GLS estimator solves the normal equations derived by differentiating $(y - X\beta)^\top \Sigma^{-1}(y - X\beta)$ with respect to $\beta$, yielding $X^\top \Sigma^{-1}(y - X\hat\beta_{\mathrm{GLS}}) = 0$, or equivalently $X^\top \Sigma^{-1} r = 0$. This is the orthogonality condition under the inner product $\langle u, v \rangle_{\Sigma^{-1}} = u^\top \Sigma^{-1} v$. Under this inner product, $\hat y$ is the projection of $y$ onto $\mathcal{C}(X)$ — meaning the residual $r$ is "perpendicular" to the column space of $X$ in the $\Sigma^{-1}$-metric sense. This confirms D is correct. Choice A describes ordinary least squares (OLS), where $\Sigma = I$ and orthogonality is Euclidean. It incorrectly applies standard projection to a setting that requires a weighted metric. Choice B introduces $\Sigma^{-2}$, which appears nowhere in the GLS derivation — this is a fabricated condition with no geometric or algebraic basis. Choice C flips the matrix: using $\Sigma$ instead of $\Sigma^{-1}$ would correspond to minimizing $(y - X\beta)^\top \Sigma(y - X\beta)$, a different problem entirely — and it incorrectly names the resulting inner product. A reliable study tip: always trace the normal equations directly from the objective function. The matrix appearing in the weight of the quadratic form is exactly the matrix that appears in the orthogonality condition $X^\top W r = 0$. Here, $W = \Sigma^{-1}$, so the answer follows immediately.
5. A full-rank model containing design matrix $X$ is augmented by one predictor $z$. Let $M_X=I-X(X^\top X)^{-1}X^\top$. Suppose $z^\top M_Xz=4$ and $z^\top M_Xy=6$. Which pair gives the coefficient of $z$ in the augmented regression and the corresponding reduction in residual sum of squares?
   Answer: The coefficient is $\frac32$, and the reduction in residual sum of squares is $9$. When a model with design matrix $X$ is augmented by a new predictor $z$, the Frisch-Waugh-Lovell (FWL) theorem tells you everything you need. The key insight is that the new coefficient and its contribution to fit depend entirely on the residualized versions of $z$ and $y$ — that is, what remains after projecting out $X$. The matrix $M_X = I - X(X^\top X)^{-1}X^\top$ is exactly this residual-maker (the annihilator of $X$). By FWL, the OLS coefficient on $z$ in the augmented model equals: $\hat{\gamma} = \frac{z^\top M_X y}{z^\top M_X z} = \frac{6}{4} = \frac{3}{2}$ The reduction in residual sum of squares (RSS) from adding $z$ — sometimes called the "extra sum of squares" — equals: $\Delta\text{RSS} = \hat{\gamma}^2 \cdot (z^\top M_X z) = \left(\frac{3}{2}\right)^2 \cdot 4 = \frac{9}{4} \cdot 4 = 9$ Equivalently, $\Delta\text{RSS} = \frac{(z^\top M_X y)^2}{z^\top M_X z} = \frac{36}{4} = 9$. This confirms A is correct. Choice B inverts the ratio for $\hat{\gamma}$, giving $2/3$, and compounds the error with an incorrect RSS. Choice C gets the coefficient right but reports the RSS as 6 — confusing $z^\top M_X y$ itself for the reduction in RSS. Choice D inverts the coefficient like B but accidentally gets the RSS right, making it a particularly tempting trap. Study tip: Memorize both FWL formulas together — $\hat{\gamma} = (z^\top M_X z)^{-1}z^\top M_X y$ and $\Delta\text{RSS} = \hat{\gamma} \cdot z^\top M_X y$ — since graduate exams routinely test whether you know both quantities and won't conflate them.
6. In a full-rank ordinary least-squares fit, the residual sum of squares using all observations is $40$. For observation $i$, the ordinary residual is $e_i=2$ and the leverage is $h_{ii}=\frac14$. The model is refitted after deleting observation $i$. What is the residual sum of squares computed from the remaining observations?
   Answer: $\frac{104}{3}$, because deletion reduces the sum by $\frac{e_i^2}{1-h_{ii}}$ after accounting for refitting. When you delete an observation from a regression model and refit, the new residual sum of squares is not simply the old RSS minus that observation's squared residual — the entire model changes. The key identity connecting full-data quantities to the deleted-model RSS is: $\text{RSS}_{(i)} = \text{RSS} - \frac{e_i^2}{1 - h_{ii}}$ This formula works because the change in fit upon deletion depends not just on how large the residual is, but on how influential the observation is via its leverage $h_{ii}$. High-leverage points "pull" the fit toward themselves, so removing them causes a larger adjustment than their raw residual suggests. Plugging in $\text{RSS} = 40$, $e_i = 2$, and $h_{ii} = \frac{1}{4}$: $\frac{e_i^2}{1 - h_{ii}} = \frac{4}{1 - \frac{1}{4}} = \frac{4}{\frac{3}{4}} = \frac{16}{3}$ $\text{RSS}_{(i)} = 40 - \frac{16}{3} = \frac{120 - 16}{3} = \frac{104}{3}$ This confirms answer B. Answer A makes the intuitive but incorrect assumption that deleting observation $i$ simply removes $e_i^2 = 4$ from the RSS, ignoring that refitting shifts all coefficient estimates. Answer C applies the wrong adjustment formula, subtracting $e_i^2 h_{ii} = 1$ instead of the correct leverage-scaled term. Answer D uses $\frac{e_i^2}{(1-h_{ii})^2}$, which is the square of the internally studentized residual structure — a plausible-looking but incorrect expression for this quantity. A useful anchor: the deleted residual $e_{(i)} = \frac{e_i}{1-h_{ii}}$ is the predicted residual for observation $i$ from the model fit without it, and $\text{RSS}_{(i)} = \text{RSS} - e_i \cdot e_{(i)}$ is the compact way to remember the correct formula.
7. Consider ordinary least squares with hat matrix $H=X(X^\top X)^{-1}X^\top$ and residual-maker $M=I-H$. The error covariance is $\sigma^2\Sigma$, where $\Sigma$ is known, symmetric, and positive definite but need not equal the identity. Which condition is necessary and sufficient for the fitted vector $Hy$ and residual vector $My$ to be uncorrelated?
   Answer: $H\Sigma=\Sigma H$, so the covariance matrix commutes with the hat matrix, leaving the model space and its complement invariant under $\Sigma$. When checking whether two random vectors are uncorrelated, you compute their cross-covariance matrix and set it to zero. Here, $\text{Cov}(Hy, My) = H \cdot \text{Cov}(y,y) \cdot M^\top = \sigma^2 H\Sigma M$. Since $H$ and $M = I - H$ are symmetric and idempotent, this simplifies to $\sigma^2 H\Sigma(I-H) = \sigma^2(H\Sigma - H\Sigma H)$. For this to be the zero matrix, you need $H\Sigma = H\Sigma H$, which is equivalent to requiring $H\Sigma M = 0$. Notice that $H\Sigma = H\Sigma H$ holds if and only if $H\Sigma(I-H) = 0$, and since $M = I - H$, this is precisely the condition $H\Sigma M = 0$. A clean way to see this: premultiplying $H\Sigma = H\Sigma H$ by nothing and postmultiplying by $H$ shows the condition is symmetric, ultimately equivalent to $H\Sigma = \Sigma H$ — confirming answer B is both necessary and sufficient. Answer A is too restrictive. Proportionality to the identity guarantees commutativity, but commutativity holds in many cases where $\Sigma \neq cI$, so A is sufficient but not necessary. Answer C concerns the structure of $X^\top \Sigma X$, which governs generalized least squares efficiency, not whether OLS fitted values and residuals are uncorrelated. Diagonality of $X^\top \Sigma X$ imposes no direct constraint on $H\Sigma M$. Answer D requires $H\Sigma H = \Sigma$, meaning $\Sigma$ lives entirely within the column space of $X$. This is far too strong — and also not the right algebraic condition derived from the cross-covariance. A key study habit: always start uncorrelation problems by writing out the cross-covariance explicitly and simplifying using the idempotency and symmetry of projection matrices before evaluating any answer choice.
8. A reduced linear model has hat matrix $H_0$ and rank $3$. A nested full model has hat matrix $H_1$ and rank $5$, based on $n=25$ observations. Suppose $y^\top(H_1-H_0)y=12$ and the full-model residual sum of squares is $40$. What is the partial $F$ statistic, including its numerator and denominator degrees of freedom?
   Answer: $F=3.00$ with degrees of freedom $2$ and $20$, using the rank difference and full-model error mean square. When comparing nested linear models, the partial $F$ statistic measures whether the additional predictors in the full model explain significantly more variance. The formula is: $F = \frac{y^\top(H_1 - H_0)y / (r_1 - r_0)}{RSS_1 / (n - r_1)}$ where $r_1 - r_0$ is the rank difference (numerator df) and $n - r_1$ is the full-model error df (denominator df). Here, $r_1 - r_0 = 5 - 3 = 2$, so the extra sum of squares must be averaged over 2 degrees of freedom: $12/2 = 6$. The full-model residual mean square is $RSS_1/(n - r_1) = 40/(25 - 5) = 40/20 = 2$. Therefore: $F = \frac{6}{2} = 3.00$ with numerator df $= 2$ and denominator df $= 20$. This confirms C is correct. Choice A fails because it places the unaveraged extra sum of squares (12) in the numerator rather than the mean square (6), inflating $F$ to 6.00 — you must divide by the numerator degrees of freedom before forming the ratio. Choice B uses denominator df $= 22$, which corresponds to $n - r_0 = 25 - 3$, the reduced model's error df. The denominator must use the full model's error df $(n - r_1 = 20)$. Choice D takes the ratio of the total extra sum of squares to the total residual sum of squares without any averaging, which is not a valid $F$ statistic construction at all. Study tip: Always track two separate averaging steps — divide the extra SS by $(r_1 - r_0)$ and the residual SS by $(n - r_1)$ — and confirm your denominator df comes from the full model, not the reduced one.
9. Let a regression design be $X=[\mathbf 1,Z]$, where $\mathbf 1$ is the intercept column. Define $P_1=\mathbf 1(\mathbf 1^\top\mathbf 1)^{-1}\mathbf 1^\top$ and $M_1=I-P_1$. Assume $M_1Z$ has full column rank. Which expression is the orthogonal projector onto $\mathcal C(X)$?
   Answer: $P_1+M_1Z(Z^\top M_1Z)^{-1}Z^\top M_1$, which projects separately onto orthogonal intercept and centered-predictor spaces. When you see a question about orthogonal projectors onto a partitioned column space, your instinct should be to apply the Frisch–Waugh–Lovell theorem and the formula for projecting onto $\mathcal{C}(X) = \mathcal{C}([\mathbf{1}, Z])$. The key idea is that you can build $P_X$ by first projecting onto $\mathcal{C}(\mathbf{1})$, then projecting the residual of $Z$ (after removing the intercept) onto its own column space. Formally, $M_1 Z$ represents the columns of $Z$ orthogonalized against $\mathbf{1}$. Since $\mathcal{C}(\mathbf{1})$ and $\mathcal{C}(M_1 Z)$ are orthogonal complements within $\mathcal{C}(X)$, the total projector decomposes as: $P_X = P_1 + P_{M_1 Z} = P_1 + M_1 Z(Z^\top M_1 Z)^{-1} Z^\top M_1$ This is exactly answer A, which correctly identifies the two orthogonal subspaces and adds their projectors — a valid operation precisely because they are orthogonal. B is wrong because $P_1 + Z(Z^\top Z)^{-1}Z^\top$ adds projectors onto non-orthogonal subspaces ($\mathbf{1} \in \mathcal{C}(Z)$ in general), so the sum is not idempotent and does not define a valid orthogonal projector. C subtracts the centered-predictor component instead of adding it, which would remove directions from the fit rather than include them — this produces neither a projector nor the correct range. D replaces $P_1$ with $M_1$, projecting onto the orthogonal complement of $\mathbf{1}$ rather than onto $\mathbf{1}$ itself, so the intercept subspace is entirely missing from the range. As a study tip: whenever the design matrix has a partitioned structure $[A, B]$, always orthogonalize $B$ against $A$ first — projectors add cleanly only when their ranges are orthogonal.

Start this quiz