Business Statistics Quiz: Model Selection And Overfitting
20 questions · exam conditions
0:00
Model Selection And OverfittingQuestion 1 of 20

An analyst compares two forecasting approaches: a simple linear trend model (y^=12.3+2.1t\hat{y} = 12.3 + 2.1t) and a complex polynomial model with interaction terms that achieves perfect fit on historical data (R2=1.00R^2 = 1.00). For the next quarter's forecast, which statement best describes the expected relative performance?

The polynomial model will outperform because it captured all historical variation and relationships in the data
The linear model will likely outperform because the polynomial model shows clear signs of overfitting
Both models will perform equally well since they were trained on the same historical dataset
The polynomial model will outperform because higher R2R^2 values indicate superior predictive capability
← Back to quizzes

Business Statistics Quiz

Business Statistics Quiz: Model Selection And Overfitting

Practice Model Selection And Overfitting in Business Statistics with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on Model Selection And Overfitting, giving you a quick way to practice the rules, question types, and explanations that matter most for Business Statistics.

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

An analyst compares two forecasting approaches: a simple linear trend model (y^=12.3+2.1t\hat{y} = 12.3 + 2.1t) and a complex polynomial model with interaction terms that achieves perfect fit on historical data (R2=1.00R^2 = 1.00). For the next quarter's forecast, which statement best describes the expected relative performance?

  1. The polynomial model will outperform because it captured all historical variation and relationships in the data
  2. The linear model will likely outperform because the polynomial model shows clear signs of overfitting (correct answer)
  3. Both models will perform equally well since they were trained on the same historical dataset
  4. The polynomial model will outperform because higher R2R^2 values indicate superior predictive capability
Explanation: A model achieving R2=1.00R^2 = 1.00 on real-world data is a strong indicator of overfitting, as it suggests the model has memorized noise rather than learned true underlying patterns. The simpler linear model is more likely to generalize better to new data. Choice A incorrectly assumes perfect historical fit translates to future performance. Choice C ignores the overfitting issue. Choice D confuses training performance with predictive ability.

Question 2

An e-commerce analyst uses the Akaike Information Criterion (AIC) to compare demand forecasting models. Model 1 has AIC = 156.3, Model 2 has AIC = 149.7, and Model 3 has AIC = 161.2. When she applies Bayesian Information Criterion (BIC), the values are 171.8, 178.4, and 169.9 respectively. If Model 2 has more parameters than the other models, what does this comparison reveal about model selection?

  1. Model 3 is optimal because it has the lowest BIC, and BIC better accounts for overfitting with many parameters (correct answer)
  2. Model 2 is optimal because it has the lowest AIC, indicating the best balance of fit and complexity
  3. The contradictory results suggest both criteria are unreliable for this particular forecasting application
  4. Model 1 provides the best compromise since it ranks second-best on both criteria consistently
Explanation: When you encounter model selection questions involving AIC and BIC, remember that these criteria balance model fit against complexity, but they penalize complexity differently. AIC applies a lighter penalty for additional parameters, while BIC imposes a heavier penalty, making it more conservative against overfitting. Looking at the data, Model 2 has the lowest AIC (149.7) but the highest BIC (178.4), while Model 3 has the lowest BIC (169.9). Since Model 2 has more parameters than the others, this pattern reveals a classic trade-off: Model 2 fits the data well (low AIC) but is heavily penalized by BIC for its complexity. Model 3 achieves the best balance according to BIC's stricter complexity penalty. Option A is correct because Model 3 has the lowest BIC, and BIC is indeed more appropriate when you're concerned about overfitting from excessive parameters. BIC's heavier penalty helps prevent selecting overly complex models that may not generalize well to new data. Option B incorrectly prioritizes AIC without considering that Model 2's superior AIC score likely comes from overfitting due to its higher parameter count. Option C wrongly assumes the criteria are unreliable when they're actually providing valuable complementary information about the fit-complexity trade-off. Option D misses the key insight that ranking "second-best" on both criteria doesn't make Model 1 optimal—the choice between AIC and BIC should depend on your tolerance for complexity. Study tip: When models have different parameter counts, favor BIC if you're more concerned about overfitting, and AIC if you prioritize predictive accuracy over simplicity.

Question 3

An analyst creates learning curves by plotting training and validation error against sample size for a sales forecasting model. The training error steadily increases while validation error decreases as sample size grows, with the two curves converging at large sample sizes. What does this pattern indicate about the current model?

  1. The model is underfitted and would benefit from additional complexity to better capture sales relationships
  2. The model is overfitted and requires regularization or complexity reduction to improve performance
  3. The model shows appropriate bias-variance balance and is likely to generalize well to new data (correct answer)
  4. The converging curves indicate the optimal sample size has been reached for this forecasting application
Explanation: When you encounter learning curves in business statistics, you're analyzing the bias-variance tradeoff and model performance. The key insight is understanding what different curve patterns reveal about your model's learning behavior. The described pattern - training error increasing while validation error decreases, with convergence at large sample sizes - indicates healthy model learning. As sample size grows, training error naturally rises because the model can't memorize a larger dataset as easily. Meanwhile, validation error drops because the model learns genuine patterns that generalize to unseen data. The convergence suggests the model has found an appropriate balance between bias and variance. Answer C correctly identifies this as a well-balanced model that should generalize effectively to new sales data. The converging curves at a reasonable error level indicate the model is capturing real relationships without overfitting. Answer A misinterprets the pattern - an underfitted model would show high error for both training and validation that remains relatively flat. Answer B confuses the scenario with overfitting, which would show very low training error but high validation error that doesn't improve with more data. Answer D incorrectly focuses on sample size optimization rather than model performance; while the curves suggest sufficient data, the question asks about model characteristics, not data adequacy. Remember this pattern: converging learning curves with decreasing validation error signal a healthy model. Watch for the opposite pattern (diverging curves with poor validation performance) as a red flag for overfitting in future problems.

Question 4

A retailer compares two demand forecasting models using time series cross-validation with expanding windows. Model A shows consistent RMSE between 12.3-13.7 across all validation windows. Model B shows RMSE ranging from 8.9-18.4 across windows, with an average of 12.8. Both models have similar complexity. Which model characteristic should most influence the selection decision?

  1. Model B should be selected because its average RMSE is lower, indicating superior overall forecasting accuracy
  2. Model B should be selected because its variable performance shows it adapts better to changing market conditions
  3. Model A should be selected because its consistent performance indicates more reliable and stable predictions (correct answer)
  4. Model A should be selected because consistent RMSE values indicate the model has avoided overfitting completely
Explanation: When evaluating forecasting models, you need to balance accuracy with reliability. While average performance matters, consistency across different time periods often proves more valuable for business decision-making. Model A demonstrates superior stability with RMSE values clustering tightly between 12.3-13.7. This narrow range indicates the model performs predictably across various market conditions and time periods. For retailers making inventory, staffing, and procurement decisions, this reliability is crucial—you can trust the model's predictions consistently. Model B's wide RMSE range (8.9-18.4) signals erratic performance. Though it occasionally achieves excellent accuracy (8.9), it also produces poor predictions (18.4). This unpredictability makes it risky for operational planning, as you can't anticipate when the model will perform well versus poorly. Answer A incorrectly prioritizes average RMSE over consistency. While Model B's average (12.8) is slightly lower, averages can mask dangerous variability. The difference is also minimal compared to the reliability gap. Answer B misinterprets volatility as adaptability. Variable performance typically indicates instability rather than sophisticated adaptation to market changes. True adaptability would show consistently good performance across different conditions. Answer D correctly identifies Model A's superiority but wrongly attributes consistent RMSE to avoiding overfitting completely. Consistent performance suggests good generalization, but doesn't guarantee the complete absence of overfitting. Study tip: In model selection questions, remember that business applications often favor reliability over marginal accuracy gains. Consistent performance enables better operational planning than occasionally brilliant but unpredictable results.

Question 5

A marketing analyst uses backward elimination with a significance threshold of p<0.05p < 0.05 to select variables for a customer lifetime value model. Starting with 15 potential predictors, the process removes 8 variables. However, when the final model is applied to predict values for a holdout sample, the mean absolute error is 23% higher than on the training data. What is the most likely explanation for this outcome?

  1. The significance threshold was too restrictive, causing important variables to be eliminated from the model
  2. The backward elimination process optimized model fit to the specific training sample, reducing generalizability (correct answer)
  3. The holdout sample contains different types of customers than those in the training dataset
  4. Seven variables are insufficient to capture the complexity of customer lifetime value relationships
Explanation: Backward elimination can lead to overfitting because the variable selection process is optimized specifically for the training data, potentially finding spurious relationships that don't generalize. The 23% increase in error on holdout data is a classic sign of overfitting from the selection process itself. Choice A is incorrect because a less restrictive threshold would likely worsen overfitting. Choice C assumes a data quality issue not suggested by the scenario. Choice D incorrectly assumes more variables would improve generalization.

Question 6

A data scientist uses stepwise regression with both forward and backward selection (αin=0.05\alpha_{in} = 0.05, αout=0.10\alpha_{out} = 0.10) to build a customer churn prediction model. The process selects 7 variables from an initial set of 20. When tested on validation data, the model's accuracy drops from 87% to 78%. What modification would most likely improve validation performance?

  1. Increase αin\alpha_{in} to 0.10 to allow more variables into the model and capture additional customer patterns
  2. Apply the same thresholds but use a larger training dataset to improve variable selection stability
  3. Use only forward selection to ensure the most important variables are identified systematically
  4. Decrease αout\alpha_{out} to 0.05 to make variable removal more stringent and reduce model complexity (correct answer)
Explanation: When you encounter stepwise regression problems showing a significant drop in validation performance (87% to 78%), you're looking at a classic overfitting scenario. The model is too complex for the available data, memorizing training patterns that don't generalize well. The correct answer is D because decreasing αout\alpha_{out} from 0.10 to 0.05 makes the removal criterion more stringent. This means variables need stronger evidence (p < 0.05 instead of p < 0.10) to stay in the model, resulting in fewer variables and reduced complexity. Since overfitting occurs when models are too complex relative to sample size, removing variables that aren't strongly significant will likely improve validation performance. Option A is wrong because increasing αin\alpha_{in} would allow even more variables into the model (requiring less statistical evidence), worsening the overfitting problem. Option B misses the point—while larger datasets can help, the fundamental issue here is model complexity, not sample size for variable selection stability. The current dataset already selected 7 variables; you need fewer, not better selection of the same number. Option C incorrectly assumes the selection method is the problem. Forward-only selection could still pick too many variables with the current thresholds, and removing the backward elimination step eliminates a useful complexity-reduction mechanism. Remember: when validation performance drops significantly below training performance in stepwise regression, think overfitting first. The solution is usually to make variable inclusion criteria more stringent (lower αin\alpha_{in}) or removal criteria less stringent (higher αout\alpha_{out}).

Question 7

A financial analyst develops a stock price prediction model using 200 observations and 45 technical indicators as predictors. The model achieves R2=0.94R^2 = 0.94 on training data. Based on the sample size to predictor ratio, what is the most appropriate next step to ensure reliable model performance?

  1. Deploy the model immediately since the high R2R^2 indicates excellent predictive capability for stock prices
  2. Apply regularization techniques like ridge or lasso regression to reduce the effective model complexity (correct answer)
  3. Increase the significance threshold for predictor inclusion to eliminate weakly significant variables
  4. Collect additional historical data to improve the model's ability to capture market patterns
Explanation: With 45 predictors and only 200 observations (ratio of 4.4:1), the model is highly susceptible to overfitting, especially given the extremely high R2R^2. Regularization techniques specifically address overfitting by penalizing model complexity. Choice A ignores the overfitting risk. Choice C (variable selection) might help but doesn't address the fundamental overfitting issue as directly as regularization. Choice D would help long-term but doesn't address the immediate overfitting problem.

Question 8

A data analytics team develops a complex regression model to predict customer churn. The model yields an R-squared of 0.95 on the training data, but when applied to a new holdout test set, the R-squared is only 0.40. Which of the following is the most likely diagnosis and the most appropriate next step?

  1. The model exhibits high bias (underfitting); the team should increase model complexity by adding polynomial terms or interaction effects.
  2. The model exhibits high variance (overfitting); the team should simplify the model by using fewer predictors or applying regularization. (correct answer)
  3. The test data is likely drawn from a different population than the training data; the team should collect a new, more representative test set.
  4. The model suffers from severe multicollinearity; the team should calculate Variance Inflation Factors (VIFs) and remove highly correlated predictors.
Explanation: The large discrepancy between the high performance on the training data (R-squared = 0.95) and the poor performance on the test data (R-squared = 0.40) is the classic sign of overfitting, also known as high variance. The model has learned the noise in the training data, not the underlying signal, and thus fails to generalize to new data. The appropriate remedy is to reduce model complexity, either by removing predictors, using a simpler functional form, or applying a regularization technique like Ridge or Lasso regression.

Question 9

A manager is presented with two predictive models for sales forecasting. Model A has a training set Mean Squared Error (MSE) of 150 and a validation set MSE of 350. Model B has a training set MSE of 220 and a validation set MSE of 280. Based on the principles of model selection for predictive accuracy, which model should be preferred and why?

  1. Model A, because its lower training MSE indicates that it provides a superior fit to the historical data.
  2. Model B, because the smaller gap between its training and validation MSE suggests it is less overfit.
  3. Model B, because its lower validation MSE suggests better predictive performance on new, unseen data. (correct answer)
  4. It is impossible to choose without knowing the number of parameters in each model to calculate AIC or BIC.
Explanation: The primary goal of model selection in a predictive context is to find the model that generalizes best to new, unseen data. The validation set MSE is the best available estimate of this out-of-sample performance. Model B has a validation MSE of 280, which is substantially lower than Model A's validation MSE of 350. Therefore, Model B is expected to be more accurate in its future predictions. While Model A's low training MSE is attractive, the large gap to its validation MSE indicates significant overfitting. While the smaller gap in Model B is a good property, the direct decision criterion is the validation performance itself.

Question 10

An analyst is building a forecasting model for quarterly sales using 10 years of data. They split the data chronologically: the first 8 years for training and the most recent 2 years for validation. They find a large gap between training and validation performance. What is a primary concern with concluding the model is overfit based solely on this evidence?

  1. The validation set is too small (only 2 years) to provide a reliable estimate of out-of-sample error.
  2. The underlying relationship between predictors and sales may have fundamentally changed in the last two years (a structural break). (correct answer)
  3. Overfitting can only be diagnosed using k-fold cross-validation, not a single chronological split.
  4. The model is likely underfit, and the poor validation performance is due to high bias from using older data.
Explanation: While the performance gap is symptomatic of overfitting, a chronological split introduces a confounder: the data-generating process itself may have changed over time. A structural break or 'concept drift' (e.g., due to a change in market conditions, consumer behavior, or competition) in the more recent validation period could cause a well-specified model to perform poorly, mimicking the symptoms of overfitting. Therefore, one cannot definitively conclude overfitting without investigating the possibility of a change in the underlying system.

Question 11

A data science team partitions their data into training, validation, and test sets. They train multiple candidate models on the training set and use the validation set to select the single best-performing model and its hyperparameters. What is the essential purpose of the final, held-out test set in this workflow?

  1. To provide a final, unbiased estimate of the chosen model's generalization performance on completely unseen data. (correct answer)
  2. To augment the training data before retraining the final model, thereby improving its accuracy.
  3. To perform a final round of hyperparameter tuning on the selected model to optimize its performance.
  4. To confirm that the variables selected by the model are statistically significant at a desired alpha level.
Explanation: The validation set is used actively in the model selection process. Because it is used to make decisions (which model to choose, which hyperparameters to use), the model selection process can indirectly 'overfit' to the validation set. The test set is held in reserve and is used only once, after all model selection and tuning is complete. Its critical purpose is to provide a final, unbiased, and realistic estimate of how the chosen model will perform in a real-world scenario on brand new data. Using it for any part of the training or selection process (choices B and C) would invalidate this purpose.

Question 12

An analyst attempts to model a complex, curvilinear business process using a simple linear regression model with only one predictor. This model produces predictions that are systematically incorrect across the range of the predictor. Which of the following best describes the error characteristics of this model?

  1. Low bias and high variance
  2. Low bias and low variance
  3. High bias and high variance
  4. High bias and low variance (correct answer)
Explanation: Bias is the error from erroneous assumptions in the learning algorithm. Here, assuming a linear relationship for a non-linear process leads to high bias; the model is fundamentally wrong and will produce systematic errors. Variance is the error from sensitivity to small fluctuations in the training set. Simple models, like the one described, are not very sensitive to the training data; if you retrained it on a different sample, the resulting line would be very similar. Thus, it has low variance. This is a classic example of an underfit model.

Question 13

A retail company builds a decision tree model to predict which customers will churn. The final tree has hundreds of specific rules, such as "If a customer is from zip code 90210, bought product X on a Tuesday, and has been a customer for 3.7 years, predict churn." The model is 99% accurate on the historical data it was trained on. Why is this model likely to be problematic for the business?

  1. The model's complexity makes it difficult for managers to understand, which will reduce their trust in its recommendations.
  2. The model has likely memorized noise and coincidences in the training data and will fail to generalize to new customers. (correct answer)
  3. The model is underfitting the data because it has not captured the true linear relationship between the predictors and churn.
  4. The use of personal data like zip codes could create legal and ethical problems for the company's marketing department.
Explanation: This is a textbook case of overfitting. A model with extremely specific rules based on a large number of predictors is likely fitting the noise and random idiosyncrasies of the training data, rather than the true underlying patterns of churn. While its historical accuracy is high, it will almost certainly perform poorly when applied to new customers whose specific details do not match these hyper-specific rules. The primary statistical problem is the failure to generalize. While interpretability (A) and data privacy (D) are valid business concerns, they are secondary to the model's fundamental predictive failure.

Question 14

When building a model to forecast a time series, such as monthly sales, a consultant argues that k-fold cross-validation is inappropriate and that a chronological split (e.g., train on years 1-8, validate on year 9) must be used. What is the strongest statistical justification for this position?

  1. K-fold cross-validation's random shuffling violates the temporal ordering of the data, allowing the model to 'see the future' and leading to an overly optimistic performance estimate. (correct answer)
  2. A chronological split is computationally much faster and requires less memory than running k iterations of model training.
  3. The independent and identically distributed (i.i.d.) assumption of regression is violated by time series, making any validation approach invalid.
  4. A chronological split results in a larger training set, which always leads to a more accurate forecasting model.
Explanation: The core problem with using standard k-fold cross-validation on time series data is information leakage. The random shuffling of data into folds means that the model, during some training iterations, will be exposed to data points from the future relative to the data points in its corresponding validation fold. This allows it to learn from future events to 'predict' past ones, something impossible in a real forecasting scenario. This leads to a performance estimate that is unrealistically good. A chronological split, or more advanced techniques like forward-chaining cross-validation, properly respects the temporal order and simulates the real-world task of predicting the future based only on the past.

Question 15

A researcher has a small dataset (n=40) and wants to build a predictive regression model from 15 candidate predictors. They use an automated forward stepwise selection procedure, which results in a model with 4 predictors and a high Adjusted R-squared. Why might this model still be unreliable for prediction?

  1. Adjusted R-squared is not an appropriate metric for small samples; cross-validated error should have been used.
  2. The stepwise procedure is likely to have capitalized on chance correlations in the small dataset, creating an overfit model. (correct answer)
  3. Forward selection is known to be inferior to backward elimination for finding the true optimal set of predictors.
  4. The final model has too many predictors relative to the sample size, violating a key assumption of linear regression.
Explanation: With small sample sizes, the risk of finding spurious relationships that exist only by chance is very high. Automated variable selection procedures like stepwise selection are particularly susceptible to this. They can easily pick predictors that appear strong in the specific sample but have no real predictive power in the broader population. This leads to an overfit model with an inflated Adjusted R-squared that will not generalize well to new data. The instability of the variable selection process is the key issue here.

Question 16

A consultant claims to have a model that considers 50 different variables to predict a company's quarterly revenue. He demonstrates that for the past 5 years of data, his model's predictions were, on average, within 0.1% of the actual revenue. Why is this high historical accuracy not a reliable indicator of the model's future forecasting ability?

  1. Using 50 variables is computationally inefficient and makes the model difficult to update each quarter.
  2. Predicting economic outcomes like revenue is inherently random, so no model can be consistently accurate.
  3. The model is likely extremely overfit, having learned the specific noise of the past 5 years rather than durable economic relationships. (correct answer)
  4. The consultant is probably misrepresenting the model's accuracy, as such a low error rate is statistically impossible.
Explanation: A model with a very large number of parameters (50 variables) relative to the number of data points (20 quarters over 5 years) has immense flexibility. This flexibility allows it to fit the historical data almost perfectly, but in doing so, it captures not only the true underlying patterns but also the random, non-repeatable noise. This is the definition of overfitting. Such a model will have excellent 'in-sample' fit but is very likely to have poor 'out-of-sample' predictive performance when faced with new data that contains different random noise.

Question 17

In a regularized regression model such as Ridge or Lasso, an analyst tunes the regularization parameter, lambda (λ\lambda). What is the conceptual effect on the bias-variance tradeoff as λ\lambda is increased from zero to a very large value?

  1. The model's variance will decrease, while its bias will increase. (correct answer)
  2. The model's variance will increase, while its bias will decrease.
  3. Both the model's bias and its variance will decrease.
  4. Both the model's bias and its variance will increase.
Explanation: The regularization parameter λ\lambda controls the strength of the penalty on the model's coefficients. When λ=0\lambda = 0, the model is equivalent to OLS and is likely to have low bias but high variance (especially with many predictors). As λ\lambda increases, the penalty becomes stronger, forcing the coefficients to shrink towards zero. This constraint makes the model less sensitive to the training data, thus decreasing its variance. However, this shrinking also pulls the coefficients away from their true values, introducing systematic error, which increases the model's bias. This is the essence of the bias-variance tradeoff in regularized models.

Question 18

A manufacturing company tracks monthly production costs and builds regression models to predict future expenses. They have 36 months of historical data and are evaluating different modeling approaches.

The company tests a polynomial time trend model that perfectly fits all 36 historical data points versus a simple linear trend that explains 89% of the variance. For forecasting month 37, which factor most strongly suggests the linear model will outperform?

  1. The linear model's mathematical simplicity makes it more computationally efficient for real-time forecasting applications
  2. Linear trends are more appropriate for manufacturing cost data due to industry-specific economic principles
  3. The 11% unexplained variance in the linear model represents systematic patterns that need complex modeling
  4. The polynomial model's perfect fit indicates it has memorized noise rather than learned true cost patterns (correct answer)
Explanation: When you encounter regression modeling questions involving model comparison, focus on the fundamental trade-off between model complexity and generalizability. A model that fits historical data perfectly often captures random noise rather than true underlying patterns. The polynomial model's perfect fit to all 36 data points is actually a red flag, not a strength. This indicates overfitting—the model has memorized every fluctuation in the historical data, including random variations that won't repeat in the future. When forecasting month 37, this overfitted model will likely perform poorly because it learned noise rather than genuine cost trends. The linear model's 89% fit suggests it captured the main pattern while ignoring random fluctuations, making it more reliable for prediction. Option A incorrectly focuses on computational efficiency, which isn't the primary concern for forecasting accuracy. Option B makes an unsupported assumption about manufacturing cost patterns—the choice between models should be based on statistical evidence, not industry generalizations. Option C misinterprets the unexplained variance as a weakness requiring complex modeling, when actually this remaining variance likely represents random noise that shouldn't be modeled. Remember this key principle: in predictive modeling, a model that explains slightly less historical variance but avoids overfitting typically forecasts better than one achieving perfect historical fit. Look for the balance between capturing true patterns and avoiding noise—perfect fits are usually too good to be true.

Question 19

An analyst compares a simple linear model and a complex random forest model using 5-fold cross-validation. The linear model has an average validation Root Mean Squared Error (RMSE) of 100 with a standard deviation across folds of 5. The random forest has an average validation RMSE of 98 with a standard deviation of 25. What is the most prudent conclusion for model selection?

  1. Choose the random forest model, as its lower average RMSE indicates it is the more accurate predictor.
  2. Choose the linear model, as its performance is far more stable and the small improvement from the complex model is likely not significant. (correct answer)
  3. Rerun the cross-validation with more folds (e.g., k=10) to confirm if the random forest's lower average RMSE is reliable.
  4. Choose the random forest model, but apply regularization techniques to reduce its high performance variance across the folds.
Explanation: While the random forest model has a slightly lower average RMSE (98 vs 100), its performance is highly variable, as shown by the large standard deviation (25 vs 5). This instability suggests that the random forest is sensitive to the particular training sample and may not be reliable. The linear model, while slightly less accurate on average in this test, is much more stable. Given that the performance difference is small and within the margin of error (a concept captured by the 'one standard error rule'), the more parsimonious and stable linear model is the more prudent choice.

Question 20

An analyst is building a multiple linear regression model. They fit an initial model and then add a new predictor variable. Which of the following statements is necessarily true regarding the effect of adding this new predictor?

  1. The Adjusted R-squared of the model will increase because the model now explains more variation in the response variable.
  2. The R-squared of the model will increase or stay the same, while the Adjusted R-squared may increase or decrease. (correct answer)
  3. Both the R-squared and the Adjusted R-squared will increase, as long as the new predictor is not perfectly collinear with existing predictors.
  4. The F-statistic for the overall model significance will increase, indicating a better model fit.
Explanation: R-squared measures the proportion of variance in the dependent variable that is predictable from the independent variable(s). By its mathematical construction, it can never decrease when a new predictor is added to the model; it will either increase (if the predictor has any correlation with the response) or stay the same (in the rare case of zero correlation). Adjusted R-squared, however, modifies R-squared by penalizing the inclusion of extra predictors. It will only increase if the new predictor improves the model by more than would be expected by chance. If the added predictor is not useful, Adjusted R-squared will decrease.