Least Squares Regression - College Statistics

A least-squares regression analysis is performed. Let $e_i = y_i - \hat{y}_i$ denote the residual for the $i$ -th observation. Which of the following quantities is NOT necessarily equal to zero?
Answer: The sum of the absolute values of the residuals, $\sum |e_i|$ . Two key properties derived from the minimization of the sum of squared errors are that the sum of the residuals is zero ( $\sum e_i = 0$ ) and the sum of the products of the explanatory variable and the residuals is zero ( $\sum x_i e_i = 0$ ). Thus, A and C must be zero. For B, if the correlation is perfect ( $r = \pm 1$ ), all points lie exactly on the line, so all residuals are zero, and the sum of their squares is also zero. However, the sum of the absolute values of the residuals, $\sum |e_i|$ , will only be zero if all residuals are zero (a perfect fit). For any imperfect fit, there will be non-zero residuals, and the sum of their absolute values will be a positive number.
In the context of least-squares regression, residuals are squared before being summed to determine the best-fitting line. What is the primary statistical reason for this specific practice?
Answer: To prevent positive and negative residuals from canceling each other out and to penalize larger errors more heavily. There are two main reasons for squaring residuals. First, if we simply summed the residuals ( $y - \hat{y}$ ), positive and negative values would cancel out, potentially resulting in a sum of zero for a line that fits poorly. Squaring makes all terms non-negative. Second, the squaring process gives more weight to larger residuals. An error of 4 contributes 16 to the sum, while two errors of 2 contribute only $4+4=8$ . This penalizes points that are far from the line more heavily than points that are close. Distractor A is partially true but misses the more important reason of penalizing large errors. C is incorrect. D is incorrect; squaring the residuals squares their units (e.g., dollars become dollars-squared), which complicates direct interpretation.
For a given dataset, Line M is the least-squares regression line. Line N is another line passing through the data such that the sum of its residuals is also zero ( $\sum(y_i - \hat{y}_{Ni}) = 0$ ). However, the sum of squared residuals for Line N is greater than for Line M. Which of the following statements must be true?
Answer: Line N must also pass through the point of means, $(\bar{x}, \bar{y})$ . When you encounter questions about regression lines and residuals, focus on the fundamental properties that define the least-squares regression line versus other lines that might fit the data. The key insight here involves understanding what it means for residuals to sum to zero. Any line where $\sum(y_i - \hat{y}_i) = 0$ must pass through the point of means $(\bar{x}, \bar{y})$ . This is a mathematical requirement: if you sum all the residuals and get zero, the line must balance above and below the center point of the data cloud. The least-squares regression line always has this property, but it's not unique to that line—other lines can also achieve this balance while having larger sums of squared residuals. Option A is incorrect because lines with zero-sum residuals don't need to be parallel; they just need to pass through $(\bar{x}, \bar{y})$ . Option B is wrong because the relationship between y-intercepts depends on the specific slopes and data—there's no requirement that Line N's y-intercept exceeds Line M's. Option C represents a common misconception: Line N can absolutely be a valid linear model even though it's not optimal. Many valid models exist for any dataset; the least-squares line is simply the best among linear models using the SSE criterion. Option D must be true because any line with residuals summing to zero mathematically must pass through the centroid of the data. Remember this principle: the sum of residuals equaling zero is equivalent to the line passing through the point of means—this connects algebraic properties to geometric intuition about regression lines.
A researcher is modeling the price of a used car ( $y$ , in dollars) based on its age ( $x$ , in years). They calculate the least-squares regression line, $\hat{y} = 25000 - 2000x$ , and find the sum of squared residuals to be 5,400,000. An alternative model, proposed by a colleague, is a simple rule of thumb: Price = 24000 - 1900x. If the researcher calculates the sum of squared residuals for the colleague's model using the same data, which of the following statements must be true?
Answer: The sum of squared residuals for the colleague's model will be greater than or equal to 5,400,000. The least-squares regression line is, by definition, the unique line that minimizes the sum of the squared residuals (SSE). The problem states that $\hat{y} = 25000 - 2000x$ is the least-squares line and its SSE is 5,400,000. Therefore, any other linear model, such as the colleague's, must have an SSE that is greater than or, in the extremely unlikely event the colleague guessed the exact same line, equal to 5,400,000. A and B contradict the definition of the LSRL. D is not necessarily true; only the LSRL is guaranteed to have a sum of residuals equal to zero.
A student is trying to find the best-fit line for a dataset. Instead of minimizing the sum of the squared vertical distances ( $\sum(y_i - \hat{y}_i)^2$ ), they mistakenly minimize the sum of the squared horizontal distances ( $\sum(x_i - \hat{x}_i)^2$ ). Which of the following is a necessary consequence of the student's procedure?
Answer: The sum of the horizontal residuals, $\sum(x_i - \hat{x}_i)$ , for the student's line will be zero. When you encounter regression problems, it's crucial to understand what minimizing different types of distances actually accomplishes. The student's approach fundamentally changes the nature of the regression problem. The correct answer is D because when you minimize horizontal distances $\sum(x_i - \hat{x}_i)^2$ , you're essentially finding the regression line of x on y (treating x as the dependent variable). Just like in standard least squares regression where $\sum(y_i - \hat{y}_i) = 0$ , this procedure guarantees that the sum of horizontal residuals equals zero. This is a mathematical consequence of the minimization process - the residuals must sum to zero when you minimize their squared values. Option A is incorrect because minimizing horizontal distances produces the regression of x on y, not y on x. These are completely different lines unless there's perfect correlation. Option B is wrong because the student's line wasn't designed to minimize vertical residuals, so it will actually have a larger sum of squared vertical residuals than the true least squares line. Option C presents a tempting misconception - while the slopes of "y on x" and "x on y" regressions are related, they're not simple reciprocals. The relationship involves correlation: if the y-on-x slope is $b$ , then the x-on-y slope is $r^2/b$ , where $r$ is the correlation coefficient. Remember this pattern: whatever type of residual you minimize (vertical or horizontal) will always sum to zero. This is a fundamental property of least squares estimation that applies regardless of which variable you treat as dependent.
The least-squares regression line for a dataset is $\hat{y} = 10 + 3x$ , and the corresponding sum of squared residuals (SSE) is 50. If the slope of the line is changed to 3.1 while the y-intercept is kept at 10, creating a new line $\hat{y}^* = 10 + 3.1x$ , what can be said about the SSE calculated for this new line?
Answer: The new SSE will be greater than 50. The least-squares regression line (in this case, $\hat{y} = 10 + 3x$ ) is the unique line that produces the minimum possible sum of squared residuals (SSE) for a given dataset. The problem states this minimum SSE is 50. Any other line, including the slightly altered line $\hat{y}^* = 10 + 3.1x$ , is not the optimal line in the least-squares sense. Therefore, when applied to the same dataset, it must result in a larger SSE. Distractors A, B, and D all contradict the fundamental definition of the least-squares line.
A least-squares regression model is fitted to a dataset with 10 observations. The residuals for the first 9 observations are: 2.5, -1.3, 0.8, -3.1, 4.0, -0.5, 1.7, -2.2, and 1.9. What is the residual for the 10th observation?
Answer: -3.8 A fundamental property of the least-squares regression line is that the sum of all residuals is equal to zero. Let the 10th residual be $e_{10}$ . We must have $\sum_{i=1}^{10} e_i = 0$ . Summing the first 9 residuals gives: 2.5 - 1.3 + 0.8 - 3.1 + 4.0 - 0.5 + 1.7 - 2.2 + 1.9 = 3.8. To make the total sum zero, $3.8 + e_{10} = 0$ , which means $e_{10} = -3.8$ . Distractor B is a calculation error (dividing the sum by 10). Distractor C is a sign error. D is incorrect because this property holds for any LSRL regardless of the data.
The relationship between the number of hours a student studies per week ( $x$ ) and their score on a final exam ( $y$ ) is modeled by the least-squares regression line $\hat{y} = 45 + 5.2x$ . One student in the study, who studied for 8 hours, scored an 82 on the exam. What is the residual for this student, and what does it indicate?
Answer: -4.6; the student scored 4.6 points lower than the model predicted for 8 hours of study. First, calculate the predicted score ( $\hat{y}$ ) for $x=8$ : $\hat{y} = 45 + 5.2(8) = 45 + 41.6 = 86.6$ . The student's actual score ( $y$ ) was 82. The residual is the actual value minus the predicted value: $y - \hat{y} = 82 - 86.6 = -4.6$ . A negative residual indicates that the observed value is below the predicted value. Therefore, the student scored 4.6 points lower than predicted by the model. Distractor B calculates the residual correctly but gets the sign wrong in the interpretation. C misinterprets the meaning of a single residual, incorrectly describing it as a rate. D is nonsensical.
An analyst fits two different linear models, Line A and Line B, to the same set of bivariate data. For Line A, the sum of the residuals, $\sum(y - \hat{y})$ , is 0, and the sum of the squared residuals, $\sum(y - \hat{y})^2$ , is 45.2. For Line B, the sum of the residuals is 2.1, and the sum of the squared residuals is 42.8. Which of the following statements is the most valid conclusion?
Answer: Line B is a better fit than Line A in the least-squares sense, as it has a smaller sum of squared residuals. The least-squares regression line (LSRL) is defined as the line that minimizes the sum of squared residuals (SSE). Line B has a smaller SSE (42.8) than Line A (45.2), meaning it is a better fit according to the least-squares criterion. While it is a property of the LSRL that the sum of its residuals is 0, this is a consequence of the minimization process, not the primary definition. The defining characteristic is the minimum possible SSE. Therefore, Line B provides a better fit. Distractor A states a true property of the LSRL but misses the more fundamental definition. C is incorrect because only SSE is minimized. D is incorrect because the LSRL is determined by minimizing SSE directly.
A scatterplot of $y$ versus $x$ shows a strong positive linear association. A least-squares regression line is fitted to the data. One data point, $(x_1, y_1)$ , has a large positive residual. If this single data point were shifted vertically downward so that its new residual (relative to the original line) became zero, and then a new least-squares line was fit to the modified data, which of the following would most likely occur?
Answer: The slope of the new regression line would decrease, and the sum of squared residuals would decrease. The point $(x_1, y_1)$ is above the regression line (large positive residual). Shifting it vertically downward reduces its y-value. This change will 'pull' the regression line toward the new, lower point. For a point with a positive residual in a positive association, this pull will typically cause the slope to decrease. The sum of squared residuals (SSE) for the new line will decrease because a point that was a major source of error has been moved to be less of an error source, reducing the overall variability of points around the fitted line. Distractor A gets the effect on the slope wrong. Distractors C and D incorrectly state that the SSE would increase, which is incorrect because the data has been modified to be less scattered.
The least-squares regression line $\hat{y} = 10.5 - 2.1x$ is calculated for a dataset. One of the data points in the set is $(x=5, y=0)$ . Which of the following statements about this data point is true?
Answer: The residual for this point is zero. To find the residual, we first calculate the predicted value $\hat{y}$ for $x=5$ . $\hat{y} = 10.5 - 2.1(5) = 10.5 - 10.5 = 0$ . The observed value is $y=0$ . The residual is $y - \hat{y} = 0 - 0 = 0$ . A residual of zero means the point lies exactly on the regression line. A is not necessarily true; since the point lies on the line, its removal may have little effect. C is incorrect; a point on the line is perfectly predicted and is the opposite of an outlier in terms of fit error. D is incorrect; for the correlation to be -1.0, all points must lie on the line, not just one.
Which of the following statements provides the most accurate description of the criterion used to find a least-squares regression line?
Answer: It is the line that minimizes the sum of the squared vertical distances from each point to the line. This question tests the precise definition of the least-squares regression line. The method specifically minimizes the sum of the squared errors. The error (or residual) is defined as the vertical distance between the observed y-value and the predicted y-value on the line ( $y - \hat{y}$ ). Distractor A is incorrect, as the line may pass through few or even no points. B describes a different, more robust type of regression (least absolute deviations regression). D describes another different method called orthogonal or total least squares regression. For the standard model taught in introductory statistics, only C is correct.
A real estate agent uses a least-squares regression model to predict the selling price of houses ( $y$ , in thousands of dollars) based on their square footage ( $x$ ). The resulting model is $\hat{y} = 50 + 0.15x$ . A particular house with 2,000 square feet sold for $340,000. What is the interpretation of the residual for this house?
Answer: The house sold for $10,000 less than its predicted price based on its size. First, calculate the predicted price for a 2,000 sq ft house. The variable $x$ represents square footage, so $x=2000$ . The predicted price (in thousands) is $\hat{y} = 50 + 0.15(2000) = 50 + 300 = 350$ . This corresponds to a predicted price of $350,000. The actual selling price was $y=340$ (in thousands). The residual is $y - \hat{y} = 340 - 350 = -10$ . Since the units are in thousands of dollars, a residual of -10 means the actual price was $10,000 less than the predicted price. Distractor A has the wrong sign. C misinterprets the residual as a rate. D misinterprets the calculation entirely.
An analyst fits a least-squares regression line to predict sales based on advertising spending. The sum of squared residuals is found to be 400 (in squared thousands of dollars). A new data point, which lies exactly on the calculated regression line, is added to the dataset. What will be the sum of squared residuals for a new least-squares line fit to this augmented dataset?
Answer: Equal to 400. Let the original least-squares regression line be $L_1$ . The sum of squared residuals for the original data with respect to $L_1$ is 400. The new data point lies exactly on $L_1$ , which means its residual relative to $L_1$ is 0. If we were to use $L_1$ as the model for the new, augmented dataset, the sum of squared residuals would be the original 400 plus the new point's contribution of $0^2$ , which is still 400. Because the new point perfectly conforms to the original trend, it does not pull the line in any new direction. Therefore, the best-fit line for the augmented dataset, $L_2$ , will be identical to $L_1$ . Since the line does not change and the new point has a zero residual, the sum of squared residuals remains 400.
A dataset has a least-squares regression line of $\hat{y} = 50 - 2x$ . The point of means, $(\bar{x}, \bar{y})$ , lies on this line. One of the points in the dataset is $(10, 35)$ . Which of the following statements is true?
Answer: The residual for the point (10, 35) is +5. First, calculate the residual for the point $(10, 35)$ . The predicted value at $x=10$ is $\hat{y} = 50 - 2(10) = 50 - 20 = 30$ . The observed value is $y=35$ . The residual is the observed value minus the predicted value: $y - \hat{y} = 35 - 30 = 5$ . Statement A is correct. Let's check the distractors. B is incorrect; while $(\bar{x}, \bar{y})$ lies on the line, we cannot assume that the single point (10, 35) is the point of means. C is incorrect; the sum of all residuals for a least-squares regression line must be 0. D is incorrect; the slope of the regression line is -2, which indicates a negative correlation, and the sign of a single residual has no bearing on the overall correlation.
For a set of 20 data points, the mean of the explanatory variable is $\bar{x} = 10$ and the mean of the response variable is $\bar{y} = 25$ . The least-squares regression line for this data is given by $\hat{y} = a + 1.5x$ . What is the value of the y-intercept, $a$ ?
Answer: 10 A required property of the least-squares regression line is that it must pass through the point of means, $(\bar{x}, \bar{y})$ . This means the coordinates of this point must satisfy the equation of the line. We can substitute $\bar{x} = 10$ and $\bar{y} = 25$ into the equation $\hat{y} = a + 1.5x$ . Since the predicted value at the mean of x is the mean of y, we get $25 = a + 1.5(10)$ . This simplifies to $25 = a + 15$ . Solving for $a$ , we find $a = 25 - 15 = 10$ . Distractor A confuses the slope with the intercept. C is the value of the term $1.5x$ . D is a calculation error ( $25+15$ ).
A researcher fits a least-squares regression line to a dataset and finds that the sum of the squared residuals is exactly zero. Which of the following is a necessary conclusion?
Answer: All of the data points lie perfectly on a straight line. The sum of squared residuals, $\sum(y_i - \hat{y}_i)^2$ , is a sum of non-negative terms. The only way for such a sum to be zero is if every individual term is zero. If $(y_i - \hat{y}_i)^2 = 0$ for all $i$ , then $y_i - \hat{y}_i = 0$ , which means $y_i = \hat{y}_i$ for every data point. This implies that every observed data point falls exactly on the regression line, meaning the data has a perfect linear relationship (correlation of +1 or -1). Distractor A is incorrect; the slope can be any value. B is also incorrect. D is a possible scenario but not a necessary one; multiple collinear points would also result in an SSE of 0.
A least-squares regression line is fit to a scatterplot of data that shows a moderate positive linear association. A new point is then added to the data. This new point has an x-value that is much larger than the other x-values in the dataset and a y-value that gives it a large positive residual with respect to the original regression line. How will a new least-squares line, fit to the augmented data, likely compare to the original line?
Answer: The slope will increase and the y-intercept will decrease. When you encounter questions about how outliers affect regression lines, focus on understanding leverage and influence. Points with extreme x-values have high leverage, meaning they can strongly pull the regression line toward themselves. Let's analyze this scenario step by step. The new point has a much larger x-value than existing data, giving it high leverage. Since it has a large positive residual from the original line, it sits well above where the original line would predict. This high-leverage point above the line will pull the new regression line upward toward itself. When the line gets pulled up toward this outlier, two things happen: the line rotates to have a steeper slope (increase), and since the line pivots around the center of the data while being pulled up on the right side, the left side drops down, decreasing the y-intercept. Think of it like a seesaw - when the right side goes up, the left side goes down. Looking at the wrong answers: Choice A incorrectly suggests both slope and y-intercept increase, missing that the line pivots downward on the left. Choice B suggests both decrease, which would happen if the outlier were below the original line. Choice C suggests the slope decreases while y-intercept increases, which contradicts how high-leverage points above the line behave. The correct answer is D - slope increases and y-intercept decreases. Study tip: Remember that high-leverage outliers pull regression lines toward themselves like magnets. Always visualize how the line would tilt: if the outlier is above the line on the right, the slope increases and y-intercept decreases.

Start this quiz