Least Squares & Normal Equations - Statistics Graduate Level

1. A simple linear regression with intercept is fit using predictor $x$. The sufficient statistics are $n=5$, $\sum x_i=10$, $\sum x_i^2=30$, $\sum y_i=20$, and $\sum x_i y_i=50$. The same fitted values are to be expressed using the centered predictor $z_i=x_i-2$ in the model $y_i=\gamma_0+\gamma_1 z_i+e_i$. What are the least squares estimates?
   Answer: $(\widehat{\gamma}_0,\widehat{\gamma}_1)=(4,1)$ When you reparametrize a regression by centering the predictor, the slope is invariant but the intercept shifts to reflect the new origin. This is the key insight: fitting $y_i = \gamma_0 + \gamma_1 z_i + e_i$ with $z_i = x_i - \bar{x}$ produces the same fitted values as the original model, so $\widehat{\gamma}_1 = \widehat{\beta}_1$, and $\widehat{\gamma}_0 = \bar{y}$ (since the regression line always passes through $(\bar{z}, \bar{y}) = (0, \bar{y})$). First, compute the sufficient statistics you need. From the given data: $\bar{x} = 10/5 = 2$, $\bar{y} = 20/5 = 4$. The slope from the original model is: $\widehat{\beta}_1 = \frac{\sum x_i y_i - n\bar{x}\bar{y}}{\sum x_i^2 - n\bar{x}^2} = \frac{50 - 5(2)(4)}{30 - 5(4)} = \frac{10}{10} = 1$ Since $z_i = x_i - 2 = x_i - \bar{x}$, the centered predictor has mean zero, so $\widehat{\gamma}_1 = \widehat{\beta}_1 = 1$ and $\widehat{\gamma}_0 = \bar{y} = 4$. The answer is (B). Choice (A) sets $\widehat{\gamma}_0 = 2 = \bar{x}$, confusing the mean of $x$ with the mean of $y$. Choice (C) sets the intercept to zero, which would only hold if $\bar{y} = 0$. Choice (D) incorrectly negates the slope, perhaps from a sign error in the centering formula. Your takeaway: centering a predictor never changes the slope — it only re-anchors the intercept to $\bar{y}$. Memorize that $\widehat{\gamma}_0 = \bar{y}$ when you center at $\bar{x}$; it's a frequent exam shortcut.
2. An existing full-rank least squares fit has $\widehat{\beta}=(1,2)^{\mathsf T}$ and $(X^{\mathsf T}X)^{-1}=I_2$. One new observation is added with predictor row $x^{\mathsf T}=(1,1)$ and response $y=5$. What is the updated least squares estimate?
   Answer: $(5/3,8/3)^{\mathsf T}$ When a single new observation is added to an existing least squares fit, you can avoid refitting from scratch using the rank-1 update formula. Given current estimate $\widehat{\beta}_{\text{old}}$, matrix $(X^\mathsf{T}X)^{-1}$, new predictor $x$, and new response $y$, the updated inverse is: $M_{\text{new}} = \left(X^\mathsf{T}X + xx^\mathsf{T}\right)^{-1}$ and the updated estimate is: $\widehat{\beta}_{\text{new}} = M_{\text{new}}\left(X^\mathsf{T}X\,\widehat{\beta}_{\text{old}} + xy\right)$ Here, $X^\mathsf{T}X = I_2$, so $X^\mathsf{T}X\,\widehat{\beta}_{\text{old}} = (1,2)^\mathsf{T}$. Adding $xy = (1,1)^\mathsf{T}\cdot 5 = (5,5)^\mathsf{T}$ gives the updated sufficient statistic $(6,7)^\mathsf{T}$. Next, $X^\mathsf{T}X + xx^\mathsf{T} = I_2 + \begin{pmatrix}1\\1\end{pmatrix}(1,1) = \begin{pmatrix}2&1\\1&2\end{pmatrix}$, whose inverse is $\frac{1}{3}\begin{pmatrix}2&-1\\-1&2\end{pmatrix}$. Multiplying: $\widehat{\beta}_{\text{new}} = \frac{1}{3}\begin{pmatrix}2&-1\\-1&2\end{pmatrix}\begin{pmatrix}6\\7\end{pmatrix} = \frac{1}{3}\begin{pmatrix}5\\8\end{pmatrix} = \left(\frac{5}{3},\frac{8}{3}\right)^\mathsf{T}$ This confirms answer A. Choice B $(3,4)^\mathsf{T}$ results from forgetting to apply $M_{\text{new}}$ and simply adding $xy$ to $\widehat{\beta}_{\text{old}}$. Choice C $(2,3)^\mathsf{T}$ comes from naively averaging the old estimate with the new observation's implied coefficients. Choice D $(4/3, 7/3)^\mathsf{T}$ arises from a sign error when inverting the updated matrix. Study tip: Memorize the rank-1 (Sherman-Morrison) update structure — on graduate exams, sequential updating questions almost always hinge on correctly computing $(X^\mathsf{T}X + xx^\mathsf{T})^{-1}$ and updating both the information matrix and the sufficient statistic simultaneously.
3. What is the residual sum of squares after adding $z$ to the model?
   Answer: $11$ When you add a new predictor $z$ to a least squares model, the new RSS equals the old RSS minus the reduction in RSS attributable to $z$'s component orthogonal to the existing column space. The key formula is: $\text{RSS}_{\text{new}} = \text{RSS}_{\text{old}} - \frac{(e^\mathsf{T}z)^2}{z^\mathsf{T}(I - P_X)z}$ Here's why: the residual vector $e = (I - P_X)y$ is already orthogonal to $\text{col}(X)$, so only the part of $z$ outside $\text{col}(X)$—namely $(I - P_X)z$—can further reduce the RSS. The reduction equals the squared correlation between $e$ and $(I-P_X)z$, normalized by the length of $(I-P_X)z$. Computing the denominator: $z^\mathsf{T}(I - P_X)z = z^\mathsf{T}z - z^\mathsf{T}P_Xz = 10 - 6 = 4$. The reduction is therefore $\frac{(e^\mathsf{T}z)^2}{z^\mathsf{T}(I-P_X)z} = \frac{2^2}{4} = \frac{4}{4} = 1$. So $\text{RSS}_{\text{new}} = 12 - 1 = 11$, confirming C is correct. Choice A (8) results from incorrectly using $z^\mathsf{T}z = 10$ as the denominator, ignoring the projection. Choice B (10) likely comes from subtracting $e^\mathsf{T}z = 2$ directly from the RSS, which has no geometric justification. Choice D (58/5) may arise from computing $12 - 4/10$, mistakenly using $z^\mathsf{T}P_Xz$ in the denominator instead of $z^\mathsf{T}(I-P_X)z$. Always remember: it's the residualized portion of $z$—not $z$ itself—that drives the reduction in RSS.
4. What are the ordinary least squares estimates of $\beta_0$ and $\beta_1$?
   Answer: $(\widehat{\beta}_0,\widehat{\beta}_1)=(8/5,\;4/5)$ When a model includes a known offset $o_i$, OLS treats it as a fixed, non-estimated component. The trick is to absorb the offset into the response by defining $y_i^* = y_i - o_i$, then regress $y_i^*$ on $x_i$ with an intercept. Your adjusted summaries become: $\sum y_i^* = \sum y_i - \sum o_i = 10 - 2 = 8$ and $\sum x_i y_i^* = \sum x_i y_i - \sum x_i o_i = 9 - 1 = 8$, with $n=4$, $\sum x_i = 2$, $\sum x_i^2 = 6$. The standard OLS normal equations give: $\widehat{\beta}_1 = \frac{n\sum x_i y_i^* - \sum x_i \sum y_i^*}{n\sum x_i^2 - (\sum x_i)^2} = \frac{4(8) - (2)(8)}{4(6) - 4} = \frac{16}{20} = \frac{4}{5}$ $\widehat{\beta}_0 = \frac{\sum y_i^* - \widehat{\beta}_1 \sum x_i}{n} = \frac{8 - \frac{4}{5}(2)}{4} = \frac{8 - 8/5}{4} = \frac{32/5}{4} = \frac{8}{5}$ This confirms answer A: $(\widehat{\beta}_0, \widehat{\beta}_1) = (8/5,\, 4/5)$. Choice B likely arises from an arithmetic error when computing $\widehat{\beta}_0$, perhaps subtracting $\sum o_i$ incorrectly from only part of the expression. Choice C comes from forgetting to subtract the offset at all — using raw $\sum y_i$ and $\sum x_i y_i$ directly. Choice D results from an entirely different misapplication, forcing the intercept to zero or misidentifying which parameters to estimate. Your go-to strategy: whenever you see an offset term, immediately redefine $y^* = y - o$ and run standard OLS on $y^*$. Don't let the notation obscure what is fundamentally a simple linear regression after that substitution.
5. In a rank-deficient least squares problem, the columns of the design matrix satisfy $X_3=X_1+X_2$. One solution of the normal equations is $b_0=(1,1,1)^{\mathsf T}$. Which coefficient vector is the minimum-Euclidean-norm least squares solution?
   Answer: $(2/3,2/3,4/3)^{\mathsf T}$ When a design matrix is rank-deficient, the normal equations have infinitely many solutions. The minimum-norm least squares solution is the unique solution with smallest Euclidean norm, given by the Moore-Penrose pseudoinverse: $b^+ = X^+y$. Critically, it lies in the row space of $X$ — orthogonal to the null space. Here, the constraint $X_3 = X_1 + X_2$ means the null space of $X$ contains the vector $n = (1, 1, -1)^{\mathsf T}$ (scaled appropriately). Specifically, any vector of the form $(1,1,-1)^{\mathsf T}t$ added to a solution still satisfies the normal equations. The general solution is $b_0 + t(1,1,-1)^{\mathsf T} = (1+t,\, 1+t,\, 1-t)^{\mathsf T}$. To find the minimum-norm solution, minimize $\|b\|^2 = (1+t)^2 + (1+t)^2 + (1-t)^2$ over $t$. Taking the derivative and setting it to zero: $2(1+t) + 2(1+t) + 2(1-t)(-1) = 0$, which gives $4(1+t) - 2(1-t) = 0 \Rightarrow 4+4t-2+2t = 0 \Rightarrow t = -1/3$. Substituting: $(1-1/3,\, 1-1/3,\, 1+1/3)^{\mathsf T} = (2/3,\, 2/3,\, 4/3)^{\mathsf T}$. That's answer B. A is wrong because $(1,1,1)^{\mathsf T}$ is merely one particular solution — it has no special minimality property. C $(4/3,4/3,2/3)^{\mathsf T}$ corresponds to $t = +1/3$, the wrong sign. D $(0,0,2)^{\mathsf T}$ uses $t = -1$, which doesn't minimize the norm. Study tip: On rank-deficient problems, always parameterize the full solution family using the null space vector, then minimize the squared norm — this one calculus step separates the minimum-norm solution from all others.
6. Which pair gives the new coefficient estimate and new residual vector?
   Answer: $(\widehat{\beta}+c,\ e+u)$ When you modify the response vector in OLS, you need to track how the projection machinery responds to each component separately. The key tool is decomposing $y^* = y + Xc + u$ and applying the hat matrix $H = X(X^\mathsf{T}X)^{-1}X^\mathsf{T}$. The new coefficient estimate is $\widehat{\beta}^* = (X^\mathsf{T}X)^{-1}X^\mathsf{T}y^*$. Expanding: $(X^\mathsf{T}X)^{-1}X^\mathsf{T}(y + Xc + u) = \widehat{\beta} + c + (X^\mathsf{T}X)^{-1}X^\mathsf{T}u$. Since $X^\mathsf{T}u = 0$, the last term vanishes, giving $\widehat{\beta}^* = \widehat{\beta} + c$. For the residuals, $e^* = y^* - X\widehat{\beta}^* = (y + Xc + u) - X(\widehat{\beta} + c) = y - X\widehat{\beta} + u = e + u$. So the answer is A: the new estimate is $\widehat{\beta} + c$ and the new residual is $e + u$. B gets the coefficient right but claims the residual is just $e$, ignoring that $u$ is orthogonal to the column space of $X$ — which means it lives entirely in the residual space and cannot be absorbed by the fitted values. C leaves $\widehat{\beta}$ unchanged, forgetting that $Xc$ is in the column space of $X$ and is fully absorbed into the coefficient estimate, not the residual. D subtracts $c$ rather than adding it, a sign error that reflects confusion about the direction of the projection update. A useful pattern to internalize: any signal in the column space of $X$ shifts the coefficients; any signal orthogonal to $X$ passes entirely into the residuals. Keeping this decomposition sharp will serve you across many OLS perturbation problems.
7. A through-the-origin model $y_i=\beta x_i+e_i$ is fit by minimizing $\sum_i w_i(y_i-\beta x_i)^2$. Two observations have $(x_1,y_1,w_1)=(1,1,4)$ and $(x_2,y_2,w_2)=(2,4,1)$. What is the weighted least squares estimate of $\beta$?
   Answer: $3/2$ Weighted least squares (WLS) generalizes ordinary least squares by giving each observation a weight that reflects its precision or reliability. When you see a WLS problem, your first instinct should be to write down the closed-form estimator rather than re-deriving it from scratch. For a through-the-origin model, minimizing $\sum_i w_i(y_i - \beta x_i)^2$ with respect to $\beta$ yields the estimator: $\hat{\beta} = \frac{\sum_i w_i x_i y_i}{\sum_i w_i x_i^2}$ This is simply the OLS formula with each term weighted by $w_i$. Plugging in the two observations: $\sum_i w_i x_i y_i = (4)(1)(1) + (1)(2)(4) = 4 + 8 = 12$ $\sum_i w_i x_i^2 = (4)(1)^2 + (1)(2)^2 = 4 + 4 = 8$ $\hat{\beta} = \frac{12}{8} = \frac{3}{2}$ This confirms D is correct. Now for the distractors. A ($6/5$) likely results from mistakenly using $\sum w_i y_i / \sum w_i x_i$, forgetting to square the $x_i$ terms in the denominator. C ($9/5$) comes from ignoring weights entirely and computing the unweighted OLS estimate $\sum x_i y_i / \sum x_i^2 = (1\cdot1 + 2\cdot4)/(1+4) = 9/5$. B ($33/17$) appears to arise from a scrambled weighting scheme, perhaps incorrectly applying weights to only the numerator or denominator but not both consistently. A reliable study tip: always write the WLS normal equations first. The formula $\hat{\beta} = (\mathbf{X}^T \mathbf{W} \mathbf{X})^{-1} \mathbf{X}^T \mathbf{W} \mathbf{y}$ reduces to this ratio in the simple no-intercept case — memorizing this structure prevents most computational errors.
8. For a full-rank least squares problem, suppose the unconstrained estimate is $\widehat{\beta}=(2,1)^{\mathsf T}$ and $X^{\mathsf T}X=\operatorname{diag}(1,4)$. The model is refit subject to the restriction $\beta_1+\beta_2=0$. What is the restricted least squares estimate?
   Answer: $(-2/5,2/5)^{\mathsf T}$ When you encounter a restricted least squares problem, your goal is to minimize the usual sum of squares subject to a linear constraint. For a constraint of the form $R\beta = r$, the restricted estimator has the closed-form solution: $\tilde{\beta} = \widehat{\beta} - (X^\mathsf{T}X)^{-1}R^\mathsf{T}\left[R(X^\mathsf{T}X)^{-1}R^\mathsf{T}\right]^{-1}(R\widehat{\beta} - r)$ Here, the constraint $\beta_1 + \beta_2 = 0$ means $R = (1,1)$ and $r = 0$. With $X^\mathsf{T}X = \text{diag}(1,4)$, its inverse is $\text{diag}(1, 1/4)$. First compute $R\widehat{\beta} - r = 2 + 1 - 0 = 3$. Then $R(X^\mathsf{T}X)^{-1}R^\mathsf{T} = 1(1) + 1(1/4) = 5/4$, so the scalar inverse is $4/5$. The correction term becomes $(X^\mathsf{T}X)^{-1}R^\mathsf{T} \cdot (4/5) \cdot 3 = \text{diag}(1,1/4)(1,1)^\mathsf{T} \cdot (12/5) = (12/5, 3/5)^\mathsf{T}$. Subtracting: $\tilde{\beta} = (2,1)^\mathsf{T} - (12/5, 3/5)^\mathsf{T} = (-2/5, 2/5)^\mathsf{T}$, confirming C. Choice A, $(1/2, -1/2)^\mathsf{T}$, satisfies the constraint but ignores the differential weighting from $X^\mathsf{T}X$ — it's what you'd get assuming an identity matrix. Choice B, $(2,-2)^\mathsf{T}$, incorrectly anchors $\beta_1$ at its unconstrained value and simply negates it. Choice D, $(-1,1)^\mathsf{T}$, satisfies the constraint but corresponds to no principled derivation given the structure of $X^\mathsf{T}X$. As a strategy, always remember that $X^\mathsf{T}X$ acts as a precision matrix — parameters estimated more precisely (larger diagonal entries) get pulled less by the constraint. Here, $\beta_2$ has four times the precision, so it shifts less, which is exactly what you see in answer C.
9. Which statement about the least squares residuals must hold, regardless of the additional predictors?
   Answer: The treated and control residual sums must each equal zero. Whenever a regression model includes an intercept and a binary group indicator $D_i$, you should immediately think about the normal equations — the system of equations OLS solves to minimize the sum of squared residuals. Each predictor in the model, including the intercept and $D_i$ itself, generates its own normal equation by requiring that the predictor is orthogonal to the residuals. The intercept term produces the equation $\sum_i \hat{e}_i = 0$, guaranteeing the overall residual sum is zero. But $D_i$ generates its own separate normal equation: $\sum_i D_i \hat{e}_i = 0$, which means the sum of residuals among treated units only equals zero. Combining these two — since $\sum_i \hat{e}_i = \sum_{D_i=0} \hat{e}_i + \sum_{D_i=1} \hat{e}_i = 0$ and $\sum_{D_i=1} \hat{e}_i = 0$ — you can immediately deduce that $\sum_{D_i=0} \hat{e}_i = 0$ as well. So both group residual sums must individually equal zero, confirming answer B. Answer A is incomplete — while the overall sum is zero, A misses the stronger constraint that each group sum is separately zero. Answer C is doubly wrong: the two group sums are not merely equal to each other but are each independently zero. Answer D is tempting because the normal equation for $D_i$ directly pins down the treated sum, but it neglects that the control sum is then forced to zero by the intercept equation. Study tip: Any time a binary group indicator is included in an OLS model with an intercept, remember that OLS perfectly fits the group means — a powerful fact that implies both group residuals sum to zero, regardless of other covariates.
10. What is the least squares coefficient of $x$ in the multiple regression?
    Answer: $14/17$ When you see a multiple regression problem with centered variables and cross-products given, your instinct should be to set up the normal equations directly. Centering eliminates the intercept from the system, so you only need to solve for the slope coefficients. The normal equations for centered variables are: $\begin{pmatrix} x^\mathsf{T}x & x^\mathsf{T}z \\ x^\mathsf{T}z & z^\mathsf{T}z \end{pmatrix} \begin{pmatrix} \hat{\beta}_x \\ \hat{\beta}_z \end{pmatrix} = \begin{pmatrix} x^\mathsf{T}y \\ z^\mathsf{T}y \end{pmatrix}$ Substituting the given values: $\begin{pmatrix} 10 & 4 \\ 4 & 5 \end{pmatrix} \begin{pmatrix} \hat{\beta}_x \\ \hat{\beta}_z \end{pmatrix} = \begin{pmatrix} 8 \\ 3 \end{pmatrix}$ The determinant is $10(5) - 4(4) = 50 - 16 = 34$. Using Cramer's rule, $\hat{\beta}_x = \frac{8(5) - 3(4)}{34} = \frac{40 - 12}{34} = \frac{28}{34} = \frac{14}{17}$, confirming answer D. Now for the distractors. Choice A ($4/5$) comes from naively dividing $x^\mathsf{T}y$ by $z^\mathsf{T}z$ — mixing up which cross-product matrix entry belongs where. Choice B ($8/5$) is the simple regression coefficient of $y$ on $x$ alone ($x^\mathsf{T}y / x^\mathsf{T}x = 8/10$... actually it equals $8/10$), or more likely from ignoring the off-diagonal terms entirely. Choice C ($5/7$) likely arises from an arithmetic error in computing the determinant or numerator, perhaps using $35/49$ from a sign mistake. The key study tip: in multiple regression, you cannot simply divide $x^\mathsf{T}y$ by $x^\mathsf{T}x$ as you would in simple regression. The off-diagonal terms in $(X^\mathsf{T}X)^{-1}$ adjust for the correlation between predictors — always solve the full system.

Start this quiz