Biostatistics Quiz: Model Selection And Overfitting
20 questions · exam conditions
0:00
Model Selection And OverfittingQuestion 1 of 20

A pharmaceutical researcher is developing a predictive model for drug response using patient genomic data. The dataset contains 500 patients and 2000 genetic markers. The researcher randomly splits the data into training (350 patients) and testing (150 patients) sets.

After fitting various models, the researcher reports that a support vector machine with radial basis function kernel achieved 92% accuracy on the test set. However, no cross-validation was performed on the training set, and the model was tuned by repeatedly testing different hyperparameters on the same test set until optimal performance was achieved. What is the primary concern with this approach?

The sample size is inadequate for reliable machine learning with 2000 features, requiring at least 10 observations per feature for stable model development.
The random split methodology is flawed for medical data, which requires stratified sampling based on patient demographics and clinical characteristics.
Support vector machines are inappropriate for genomic data due to the high-dimensional nature of genetic markers and potential multicollinearity issues.
The test set has been contaminated by hyperparameter tuning, making the reported 92% accuracy an overly optimistic estimate of true generalization performance.
← Back to quizzes

Biostatistics Quiz

Biostatistics Quiz: Model Selection And Overfitting

Practice Model Selection And Overfitting in Biostatistics with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on Model Selection And Overfitting, giving you a quick way to practice the rules, question types, and explanations that matter most for Biostatistics.

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

A pharmaceutical researcher is developing a predictive model for drug response using patient genomic data. The dataset contains 500 patients and 2000 genetic markers. The researcher randomly splits the data into training (350 patients) and testing (150 patients) sets.

After fitting various models, the researcher reports that a support vector machine with radial basis function kernel achieved 92% accuracy on the test set. However, no cross-validation was performed on the training set, and the model was tuned by repeatedly testing different hyperparameters on the same test set until optimal performance was achieved. What is the primary concern with this approach?

  1. The sample size is inadequate for reliable machine learning with 2000 features, requiring at least 10 observations per feature for stable model development.
  2. The random split methodology is flawed for medical data, which requires stratified sampling based on patient demographics and clinical characteristics.
  3. Support vector machines are inappropriate for genomic data due to the high-dimensional nature of genetic markers and potential multicollinearity issues.
  4. The test set has been contaminated by hyperparameter tuning, making the reported 92% accuracy an overly optimistic estimate of true generalization performance. (correct answer)
Explanation: When evaluating machine learning models, you must distinguish between different types of data "leakage" that can inflate performance estimates. This question tests your understanding of proper validation methodology and how test set contamination occurs. The fundamental problem here is that the researcher used the test set multiple times during model development. By repeatedly testing different hyperparameters on the same test set and selecting the combination that performed best, the researcher essentially trained on the test data. This process means the model has "seen" the test set during development, making it no longer an independent measure of generalization performance. The reported 92% accuracy is artificially inflated because the hyperparameters were optimized specifically for this test set. This is why answer D is correct. Let's examine why the other options miss the mark: A incorrectly applies a rigid "10 observations per feature" rule that doesn't account for regularization techniques and dimension reduction methods commonly used with high-dimensional data. B misidentifies the splitting methodology as the primary issue—while stratified sampling can be beneficial, random splitting is generally acceptable and isn't the main problem here. C makes an unfounded claim about SVM inappropriateness for genomic data; SVMs with RBF kernels are actually well-suited for high-dimensional problems and can handle multicollinearity effectively. Remember this key principle: your test set should only be used once, at the very end of model development. Use cross-validation on your training set for hyperparameter tuning, then evaluate final performance on a truly held-out test set.

Question 2

A researcher builds three nested regression models to predict blood pressure: Model 1 has 2 predictors, Model 2 has 5 predictors, and Model 3 has 12 predictors. The R-squared values are 0.45, 0.62, and 0.89 respectively. If the sample size is 30 subjects, which statement best describes the likely validity of these results?

  1. Model 3 is clearly the best choice since it has the highest R-squared value
  2. Model 2 provides the optimal balance between explanatory power and model complexity
  3. Model 3 likely suffers from overfitting due to insufficient observations per predictor (correct answer)
  4. All models are equally valid since they use the same dataset for comparison
  5. Model 1 should be rejected because its R-squared is below the acceptable threshold of 0.50
Explanation: When evaluating regression models with increasing complexity, you need to consider the relationship between sample size, number of predictors, and model reliability. A critical rule of thumb is having at least 10-15 observations per predictor variable to avoid overfitting. With only 30 subjects, Model 3's 12 predictors creates a problematic ratio of 2.5 observations per predictor. This severely violates the minimum sample size requirements and makes the model prone to overfitting—where the model memorizes noise in the training data rather than learning generalizable patterns. The suspiciously high R-squared of 0.89 is a red flag indicating the model is likely fitting to random variation rather than true relationships. Choice A falls into the common trap of assuming higher R-squared automatically means better model performance. While Model 3 has the highest R-squared, this metric becomes misleading with insufficient sample sizes. Choice B suggests Model 2 is optimal, but with 6 observations per predictor, it's still underpowered, though less severely than Model 3. Choice D incorrectly assumes all models are equally valid simply because they use the same dataset—sample size adequacy depends on model complexity, not just data source. Remember this key principle: R-squared will always increase (or stay the same) as you add predictors, regardless of whether those predictors are meaningful. In small samples, this increase often reflects overfitting rather than improved model quality. Always check your observations-to-predictors ratio before interpreting R-squared values.

Question 3

A biostatistician uses 10-fold cross-validation to evaluate a logistic regression model predicting disease outcome. The average training accuracy across folds is 94%, while the average validation accuracy is 67%. The difference between these accuracies primarily indicates:

  1. The cross-validation procedure was implemented incorrectly across the folds
  2. The model demonstrates excellent generalizability to new populations
  3. The sample size is too small for reliable cross-validation assessment
  4. The model is severely overfitted and has poor generalization ability (correct answer)
  5. The 10-fold approach is inappropriate for logistic regression validation
Explanation: When evaluating machine learning models, cross-validation helps assess how well your model will perform on unseen data by comparing training performance (how well the model fits the data it learned from) to validation performance (how well it performs on held-out data). A large gap between these metrics is a classic red flag. The correct answer is D because the 27 percentage point difference (94% vs 67%) indicates severe overfitting. When a model achieves very high training accuracy but much lower validation accuracy, it has essentially "memorized" the training data rather than learning generalizable patterns. This means it will perform poorly on new, unseen data—exactly what we see here. Let's examine why the other options are incorrect. Option A suggests implementation error, but cross-validation showing consistent patterns across folds (high training, lower validation accuracy) actually indicates the procedure is working correctly to detect overfitting. Option B claims excellent generalizability, which is the opposite of what these results show—poor validation performance means poor generalizability. Option C blames sample size, but overfitting can occur regardless of sample size and is more related to model complexity relative to the data. Study tip: Remember that in model validation, you want training and validation accuracies to be close to each other. A large gap (especially >15-20 percentage points) almost always signals overfitting. When you see high training performance paired with poor validation performance on biostatistics exams, think "overfitting" immediately.

Question 4

In forward stepwise regression, variables are added sequentially based on statistical significance. If a researcher stops adding variables when p-values exceed 0.05, but then finds the final model performs poorly on new data despite having all significant predictors, what is the most likely explanation?

  1. The 0.05 significance threshold was too conservative for variable selection
  2. Forward stepwise regression inherently produces biased coefficient estimates
  3. The model selection process capitalized on chance relationships in the sample (correct answer)
  4. The new dataset has different outcome variable definitions than the original
  5. Stepwise regression requires backward elimination to achieve optimal results
Explanation: When you encounter questions about model selection procedures like stepwise regression, focus on the distinction between statistical significance and predictive validity. These are fundamentally different concepts that can lead to very different outcomes. The correct answer is C because forward stepwise regression creates a multiple testing problem. When you sequentially test many variables for inclusion, you're essentially conducting numerous hypothesis tests on the same dataset. Even if individual variables meet the 0.05 significance threshold, the process capitalizes on random fluctuations in your sample data. Variables that appear significant due to chance relationships will fail to replicate in new data, leading to poor out-of-sample performance despite having a model full of "significant" predictors. Option A is incorrect because using a more liberal threshold (say, 0.10) would actually worsen the overfitting problem by including even more chance relationships. Option B misses the mark - while stepwise regression has limitations, the specific issue here isn't about coefficient bias but about variable selection capitalizing on noise. Option D introduces an external validity concern, but the question describes poor performance on "new data," implying the same type of outcome measurement. The key insight is that statistical significance in your training sample doesn't guarantee real predictive relationships. This phenomenon, sometimes called "selection bias" or "winner's curse," is why cross-validation and holdout testing are crucial in model development. Study tip: Remember that p-hacking can happen inadvertently through automated procedures. Always validate model performance on truly independent data, regardless of how many significant predictors you found.

Question 5

A researcher compares three model selection approaches: (1) using all available predictors, (2) forward selection with p < 0.05, and (3) LASSO regression with cross-validated lambda. When evaluated on holdout data, approach (3) has the lowest prediction error despite having moderate R-squared on the training data. This result suggests:

  1. LASSO regression produces biased estimates that artificially improve validation performance
  2. The other approaches suffered from overfitting while LASSO achieved better bias-variance balance (correct answer)
  3. Cross-validated lambda selection was inappropriate for this particular dataset
  4. R-squared is a poor measure of model quality compared to prediction error
  5. The holdout data evaluation was contaminated by information from the training process
Explanation: When you encounter model comparison scenarios in biostatistics, focus on the fundamental bias-variance tradeoff that governs predictive performance. Models can fail in two key ways: bias (systematic error from oversimplified assumptions) or variance (excessive sensitivity to training data fluctuations). The scenario describes a classic overfitting situation. Using all predictors (approach 1) likely captured noise along with signal, while forward selection with p < 0.05 (approach 2) can be notoriously unstable and prone to selecting spurious variables. Both methods probably achieved high training R-squared by fitting to dataset-specific patterns that don't generalize. LASSO regression with cross-validated lambda selection (approach 3) automatically shrinks coefficients toward zero and excludes irrelevant predictors, achieving better bias-variance balance. The moderate training R-squared actually signals appropriate regularization rather than poor performance. Answer A incorrectly suggests LASSO produces harmful bias. While LASSO does introduce bias through shrinkage, this bias is beneficial when it prevents overfitting. Answer C misinterprets the results - cross-validated lambda selection worked exactly as intended by optimizing generalizability rather than training fit. Answer D creates a false opposition between R-squared and prediction error. R-squared remains valuable for training data assessment, but prediction error on holdout data is indeed the gold standard for model comparison. Remember this pattern: when regularized methods show modest training performance but superior validation performance, they're successfully controlling overfitting. High training R-squared with poor holdout performance is a red flag for overfitting, not a sign of model quality.

Question 6

In a study with 50 participants, a researcher creates polynomial terms up to degree 6 for age (age, age², age³, age⁴, age⁵, age⁶) and includes all terms in a regression model predicting cognitive score. The model achieves R² = 0.87. What is the primary concern with this modeling approach?

  1. Polynomial terms of degree 6 violate the linearity assumption of regression
  2. The high R² value indicates the presence of influential outliers in the data
  3. Six polynomial terms create excessive model complexity relative to sample size (correct answer)
  4. Age should be log-transformed rather than using polynomial expansion
  5. Cognitive scores require non-parametric modeling approaches instead of regression
Explanation: When you encounter regression problems involving many predictor variables relative to sample size, immediately consider the risk of overfitting. This occurs when a model becomes too complex for the available data, fitting noise rather than true underlying relationships. With 50 participants and 6 polynomial terms for age, you're using 12% of your sample size just on variations of a single variable. This creates excessive model complexity that likely explains the suspiciously high R² = 0.87. The model is probably memorizing random fluctuations in this small dataset rather than capturing genuine patterns that would generalize to new data. Option A is incorrect because polynomial terms don't violate linearity assumptions—regression assumes linearity in the parameters, not the variables themselves. Polynomial terms like age² are still linear in their coefficients. Option B misinterprets what high R² indicates. While outliers can inflate R², the primary red flag here is overfitting from too many predictors, not outlier influence. Option D suggests a different transformation approach but misses the core issue. The problem isn't the type of transformation chosen, but rather including too many terms regardless of whether they're polynomial, logarithmic, or any other form. Remember the general guideline: you need at least 10-15 observations per predictor variable to avoid overfitting. When you see high R² values with complex models and small samples, suspect overfitting rather than celebrating the model's apparent performance.

Question 7

A biostatistician splits data into training (70%) and test (30%) sets, then uses the training set for both model building and hyperparameter tuning through repeated testing of different combinations. The final model shows excellent performance on the test set. What methodological concern should be raised?

  1. The 70/30 split ratio is inappropriate for this type of analysis
  2. Hyperparameter tuning should be performed on the entire dataset before splitting
  3. The test set performance may be overly optimistic due to indirect overfitting (correct answer)
  4. Training set size is insufficient for reliable hyperparameter optimization
  5. Multiple hyperparameter combinations require Bonferroni correction for significance
Explanation: When you encounter questions about model validation and data splitting, focus on the fundamental principle that your test set must remain completely untouched during model development to provide an unbiased performance estimate. The key issue here is indirect overfitting or data leakage. While the biostatistician didn't directly train on the test set, they used it repeatedly to evaluate different hyperparameter combinations during tuning. Each time you check test set performance to guide your decisions, you're essentially incorporating information from that test set into your model selection process. This means your final model has been indirectly optimized for this specific test set, making the performance estimate overly optimistic and unreliable for truly unseen data. Looking at the wrong answers: (A) The 70/30 split ratio is actually quite standard and appropriate for most analyses. (B) suggests using the entire dataset for hyperparameter tuning before splitting, which would create massive data leakage and make any subsequent validation meaningless. (D) focuses on training set size, but 70% of the data is typically sufficient for hyperparameter optimization in most scenarios. The correct approach requires a three-way split: training (for model fitting), validation (for hyperparameter tuning), or using techniques like cross-validation within the training set only, keeping the test set completely separate until final evaluation. Study tip: Remember the golden rule of model validation: your test set should be opened only once, at the very end, like Schrödinger's box. Any repeated peeking compromises its validity.

Question 8

Two regression models predicting hospital readmission are compared: Model X uses 4 carefully selected clinical variables, while Model Y uses 25 variables including many demographic and administrative factors. Both models have similar AIC values. When choosing between these models for clinical implementation, what factor should be prioritized?

  1. Model Y should be chosen because it captures more comprehensive patient information
  2. Model X should be chosen due to its superior interpretability and reduced overfitting risk (correct answer)
  3. The models are equivalent since AIC values are similar, so either can be used
  4. Additional model validation using BIC instead of AIC is needed for proper comparison
  5. Model Y provides better coverage of confounding variables and should be preferred
Explanation: When comparing predictive models for clinical implementation, you need to balance statistical performance with practical considerations like interpretability, overfitting risk, and implementation feasibility. Similar AIC values indicate comparable model fit, but this doesn't make the models equivalent for real-world use. Model X's superior choice stems from the principle of parsimony and practical clinical utility. With only 4 carefully selected clinical variables, it offers several advantages: clinicians can easily understand and trust the model's predictions, fewer variables reduce the risk of overfitting (especially important with limited sample sizes), and implementation requires collecting less data. Complex models with many variables often capture noise rather than true signal, making them less generalizable to new patients. Option A incorrectly assumes that more variables automatically improve model quality. While comprehensive information seems valuable, Model Y's 25 variables likely include redundant or weakly predictive factors that add complexity without meaningful benefit. Option C misunderstands that similar AIC values don't make models clinically equivalent—implementation considerations matter enormously. Option D suggests BIC validation would change the fundamental comparison, but BIC actually penalizes complex models more heavily than AIC, likely favoring the simpler Model X even more strongly. Remember this key principle: in clinical prediction modeling, when statistical performance is similar, choose the simpler model. Simpler models are more interpretable, less prone to overfitting, easier to implement, and more likely to maintain performance when applied to new patient populations.

Question 9

A medical researcher uses ridge regression with cross-validation to select the regularization parameter λ. The optimal λ shrinks several coefficient estimates to near zero but doesn't eliminate them entirely. Compared to ordinary least squares regression, this result indicates:

  1. The ridge regression model is underfitted and should use a smaller λ value
  2. Ridge regression has successfully balanced bias and variance to improve generalization (correct answer)
  3. The near-zero coefficients indicate these variables should be completely removed
  4. Cross-validation failed to identify the correct λ value for this dataset
  5. Ridge regression is inappropriate when variables have different measurement scales
Explanation: When you encounter ridge regression questions, focus on understanding the bias-variance tradeoff and how regularization affects model performance. Ridge regression adds a penalty term that shrinks coefficients toward zero to prevent overfitting. The described scenario perfectly illustrates ridge regression working as intended. Cross-validation identified an optimal λ that shrinks coefficients to near-zero values, which means the model has found the sweet spot between fitting the training data and maintaining good generalization performance. This shrinkage introduces a small amount of bias but significantly reduces variance, typically leading to better performance on new data than ordinary least squares. Let's examine why the other options miss the mark. Option A incorrectly suggests underfitting - if the model were underfitted, you'd want more regularization (larger λ), not less. A smaller λ would reduce shrinkage and potentially lead to overfitting. Option C reflects a common misconception about ridge regression: unlike LASSO regression, ridge never eliminates variables entirely. The near-zero coefficients are actually the desired outcome, showing these variables have minimal predictive value while still being retained in the model. Option D wrongly assumes cross-validation failed when it actually succeeded in finding the optimal balance point. Study tip: Remember that ridge regression shrinks but never eliminates, while LASSO can set coefficients to exactly zero. When you see "near-zero but not eliminated" in a question, think "ridge regression working correctly." The goal isn't to remove variables but to find the optimal bias-variance balance for better generalization.

Question 10

A pharmaceutical researcher is developing a model to predict drug efficacy based on patient characteristics. The dataset contains 100 patients with 15 potential predictors including age, weight, genetic markers, and lab values.

The researcher fits a full model with all 15 predictors and achieves R² = 0.89 with all predictors showing p < 0.05. However, when the model is applied to a new cohort of 50 patients, the prediction accuracy drops significantly. Based on this scenario, what is the most appropriate next step?

  1. Increase the significance threshold to p < 0.01 to ensure only the strongest predictors remain
  2. Collect additional patients for the original dataset to improve the model stability
  3. Implement regularization techniques or reduce model complexity to address overfitting (correct answer)
  4. Conclude that the model is invalid due to population differences between cohorts
  5. Re-fit the model using the combined dataset of 150 patients for better power
Explanation: When you encounter a scenario where a model performs excellently on training data but poorly on new data, you're dealing with a classic overfitting problem. This is especially common when the ratio of predictors to observations is high, as in this case (15 predictors for 100 patients). The dramatic drop in prediction accuracy on the new cohort, despite the impressive R² = 0.89 and significant p-values in the original model, strongly indicates that the model learned noise specific to the training data rather than generalizable patterns. Choice C correctly identifies this as an overfitting issue requiring regularization techniques (like LASSO or ridge regression) or model simplification to improve generalizability. Choice A misunderstands the problem—changing the significance threshold won't address overfitting and could actually worsen it by focusing on spuriously significant predictors. Choice B suggests collecting more data for the original dataset, but this doesn't solve the fundamental issue of having too many predictors relative to sample size, and the model has already been built. Choice D incorrectly assumes population differences are the cause, but the more likely explanation is model complexity given the high predictor-to-sample ratio. Remember this key biostatistics principle: when you see high training performance but poor validation performance, especially with many predictors relative to sample size, think overfitting first. The solution involves reducing model complexity, not tweaking significance levels or assuming external factors.

Question 11

In comparing nested models using likelihood ratio tests, Model A (3 parameters) and Model B (8 parameters) both fit the same dataset. The likelihood ratio test shows p = 0.02, favoring Model B. However, when evaluated using 5-fold cross-validation, Model A shows consistently lower prediction error. How should these conflicting results be interpreted?

  1. The likelihood ratio test is more reliable since it's based on the full dataset
  2. Model B's additional parameters improve fit but may not enhance generalization (correct answer)
  3. Cross-validation results are unreliable due to the small number of folds used
  4. The models are not properly nested, invalidating the likelihood ratio test
  5. Statistical significance always takes precedence over cross-validation performance
Explanation: When you encounter questions about model comparison that involve both statistical significance tests and predictive performance measures, you're dealing with the fundamental tension between model fit and generalizability—a core concept in biostatistics. The likelihood ratio test with p = 0.02 correctly indicates that Model B (8 parameters) fits the current dataset significantly better than Model A (3 parameters). This makes statistical sense: more parameters typically allow a model to capture more patterns in the data. However, the cross-validation results tell a different story about how well each model performs on unseen data. Model A's consistently lower prediction error suggests it generalizes better, even though it fits the training data less perfectly. This scenario illustrates overfitting: Model B's additional 5 parameters improve fit to the specific dataset but capture noise rather than true underlying relationships. Answer B correctly identifies this phenomenon—the extra parameters enhance fit but harm generalization. Answer A is wrong because statistical significance doesn't automatically mean better practical performance. The full dataset perspective matters for fit, but generalization requires different evaluation. Answer C incorrectly dismisses cross-validation; 5-fold cross-validation is actually quite standard and reliable for model comparison. Answer D misunderstands the scenario—Model A is nested within Model B (fewer parameters), making the likelihood ratio test perfectly valid. Remember this pattern: when fit-based tests (like likelihood ratio) contradict prediction-based tests (like cross-validation), suspect overfitting. The simpler model often wins in real-world applications where generalization matters more than perfect fit.

Question 12

When using information criteria (AIC, BIC) for model selection, BIC tends to select simpler models than AIC when sample sizes are large. A researcher finds that AIC selects a 12-parameter model while BIC selects a 6-parameter model from the same candidate set. For minimizing overfitting risk, which approach is generally preferable?

  1. AIC is preferable because it maximizes model explanatory power
  2. BIC is preferable because it penalizes complexity more heavily (correct answer)
  3. The choice should depend on the specific research objectives rather than overfitting concerns
  4. Both criteria are equally effective at preventing overfitting in large samples
  5. A combined approach using both AIC and BIC simultaneously should be implemented
Explanation: When you encounter questions about information criteria for model selection, focus on understanding how AIC and BIC balance model fit against complexity, and how their penalty structures differ. BIC (Bayesian Information Criterion) applies a stronger penalty for model complexity than AIC (Akaike Information Criterion), especially as sample size increases. The BIC penalty term includes ln⁡(n)\ln(n) where n is sample size, while AIC uses a constant penalty of 2. This means BIC becomes increasingly conservative about adding parameters as your dataset grows, making it more effective at preventing overfitting in large samples. Option B is correct because BIC's heavier complexity penalty directly addresses overfitting risk. When BIC selects a 6-parameter model over AIC's 12-parameter choice, it's prioritizing parsimony and generalizability over maximum fit to the current data. Option A is wrong because maximizing explanatory power often increases overfitting risk rather than minimizing it. A model that explains more variance in your sample may perform poorly on new data. Option C misses the point—while research objectives matter for many modeling decisions, the question specifically asks about overfitting risk, where BIC's conservative approach has a clear theoretical advantage. Option D incorrectly suggests equivalent performance. In large samples, AIC's tendency toward more complex models actually increases overfitting risk compared to BIC's more stringent selection. Study tip: Remember that BIC = "Be Increasingly Conservative"—its logarithmic sample size penalty makes it favor simpler models more strongly as n grows, which is exactly what you want for overfitting prevention.

Question 13

A logistic regression model for disease prediction achieves 95% accuracy on training data but only 72% accuracy on test data. The model includes 15 predictors with a training sample of 200 patients. Which modification would most likely improve the generalization performance?

  1. Increase the sample size to achieve better parameter estimation stability
  2. Apply feature selection to reduce the number of predictors in the model (correct answer)
  3. Transform all continuous predictors using polynomial terms for better fit
  4. Use a different classification threshold to optimize test set accuracy
  5. Replace logistic regression with a more flexible non-parametric approach
Explanation: When you encounter a scenario where training accuracy far exceeds test accuracy (95% vs 72%), you're looking at a classic case of overfitting. The model has memorized the training data rather than learning generalizable patterns, especially problematic here with 15 predictors and only 200 patients—that's roughly 13 observations per predictor, well below recommended ratios. Option B is correct because feature selection directly addresses overfitting by removing irrelevant or redundant predictors. With fewer parameters to estimate, the model becomes more parsimonious and generalizable. This reduces the model's capacity to memorize training-specific noise while preserving important predictive relationships. Option A, while generally beneficial, won't solve the immediate overfitting problem. The issue isn't just sample size but the predictor-to-sample ratio. Adding more data helps, but reducing predictors is more direct and often more feasible. Option C would worsen the problem dramatically. Polynomial transformations increase model complexity by creating additional parameters, making overfitting even more severe. This is exactly the opposite of what's needed. Option D addresses a different issue entirely. Threshold optimization can improve classification performance metrics but doesn't fix the underlying overfitting problem. The poor generalization stems from model complexity, not classification cutoffs. Remember this pattern: when training performance greatly exceeds test performance, especially with high predictor-to-sample ratios, think overfitting first. The solution typically involves reducing model complexity through feature selection, regularization, or simpler model architectures rather than increasing complexity or just tweaking outputs.

Question 14

A researcher uses 10-fold cross-validation to compare multiple regression models with different numbers of predictors. Model A (5 predictors) has a mean CV error of 2.3 with standard deviation 0.8. Model B (12 predictors) has a mean CV error of 2.1 with standard deviation 1.4. Based on these results, which model should be selected?

  1. Model B because it has the lowest mean cross-validation error
  2. Model A because the difference in mean error is not practically significant
  3. Model A because it combines lower complexity with more consistent performance (correct answer)
  4. Model B because more predictors provide better population representation
  5. Additional cross-validation runs are needed to determine statistical significance
Explanation: When comparing machine learning models using cross-validation, you need to consider both predictive accuracy and model complexity. Cross-validation helps estimate how well models will generalize to new data, but the mean error is just one piece of the puzzle. Model A demonstrates superior performance characteristics despite its slightly higher mean error. With 5 predictors versus Model B's 12, it achieves nearly identical predictive performance (2.3 vs 2.1 mean CV error) while being much simpler. More importantly, Model A's standard deviation of 0.8 compared to Model B's 1.4 indicates much more consistent performance across different data subsets. This consistency suggests Model A will be more reliable when applied to new data. Option A is incorrect because selecting models based solely on the lowest mean error ignores the bias-variance tradeoff. The tiny difference in mean error (0.2) doesn't justify the added complexity and inconsistency. Option B misses the point entirely—this isn't about practical significance of the error difference, but about balancing multiple model characteristics. Option D reflects a common misconception that more predictors automatically improve generalization; in reality, additional predictors often lead to overfitting and reduced reliability. Study tip: When evaluating cross-validation results, always examine three factors together: mean performance, consistency (standard deviation), and model complexity. A model that performs nearly as well with fewer parameters and more consistent results typically generalizes better than a complex model with marginally better but highly variable performance.

Question 15

A biostatistician builds a survival model using Cox regression with 25 covariates and 150 events. The model shows excellent concordance (C-index = 0.89) on the training data. However, when applied to an external validation cohort, the C-index drops to 0.61. What ratio of events to variables contributed to this performance degradation?

  1. The 6:1 ratio of events to variables is adequate and cannot explain the poor validation
  2. The 6:1 ratio is below recommended guidelines and likely caused overfitting (correct answer)
  3. The 25:150 ratio indicates insufficient variable representation in the model
  4. External validation requires the same events-to-variables ratio as internal validation
  5. Cox regression performance is independent of the events-to-variables ratio
Explanation: When you encounter Cox regression problems involving model performance differences between training and validation data, think about the events-per-variable (EPV) ratio and overfitting risk. This is a classic scenario where inadequate sample size relative to model complexity causes poor generalizability. Let's calculate the EPV ratio: 150 events ÷ 25 covariates = 6 events per variable. The widely accepted guideline for Cox regression recommends at least 10-15 events per variable to avoid overfitting. With only 6 EPV, this model is severely underpowered. The dramatic drop from C-index 0.89 to 0.61 between training and validation is a textbook sign of overfitting - the model memorized noise in the training data rather than learning generalizable patterns. Option A incorrectly claims 6:1 is adequate when established guidelines require much higher ratios. Option C misinterprets the ratio direction - it's not about "variable representation" but rather having sufficient events to reliably estimate each variable's effect. Option D makes no sense because external validation doesn't require matching the training EPV ratio; it tests whether your model generalizes regardless of the validation cohort's composition. Option B correctly identifies that the 6:1 ratio falls below recommended guidelines and explains the overfitting that caused poor external validation performance. Study tip: Remember the "10-15 EPV rule" for Cox regression. When you see high training performance that crashes during validation, immediately check if there are enough events per variable. This is one of the most common causes of model failure in survival analysis.

Question 16

Two research teams analyze the same dataset differently: Team A uses all available predictors in their model, while Team B uses penalized regression with cross-validated regularization. Both teams report their results using the same test set. Team A reports R² = 0.78, Team B reports R² = 0.66. Which team's approach provides more trustworthy evidence of model generalizability?

  1. Team A because their higher R² demonstrates superior model performance
  2. Team B because regularization provides built-in protection against overfitting (correct answer)
  3. Both approaches are equally trustworthy since they use the same test set
  4. Neither approach is trustworthy because the teams analyzed the same data
  5. Team A because using all predictors captures the complete data structure
Explanation: When you encounter questions about model validation and generalizability, focus on which approach better guards against overfitting—the tendency for models to memorize training data rather than learn generalizable patterns. Team B's penalized regression approach is more trustworthy because regularization techniques like LASSO or Ridge regression systematically prevent overfitting by penalizing model complexity. Cross-validated regularization adds another layer of protection by optimizing the penalty strength using multiple validation folds, ensuring the model doesn't just perform well on one particular data split. While Team B's lower R² might seem disappointing, it likely represents a more realistic estimate of true predictive performance. Option A falls into the common trap of equating higher R² with better models. Team A's approach—using all available predictors without regularization—is a recipe for overfitting, especially with limited sample sizes. Their impressive R² = 0.78 may largely reflect memorization of noise rather than genuine predictive ability. Option C misses the crucial point that while both teams used the same test set, their model development processes differed dramatically in their susceptibility to overfitting. The test set alone doesn't guarantee trustworthy results if the model was overfit during training. Option D incorrectly suggests that analyzing the same dataset is problematic—replication using identical data is actually valuable for comparing methodological approaches. Remember: In biostatistics, a slightly lower performance metric from a properly regularized model often indicates better generalizability than impressive numbers from an overfit model. Always prioritize methodological rigor over flashy performance statistics.

Question 17

A researcher compares model selection strategies: Strategy 1 uses p-value thresholds, Strategy 2 uses AIC comparison, and Strategy 3 uses cross-validated prediction error. When applied to the same dataset, the three strategies select models with 8, 12, and 5 variables respectively. For ensuring model generalizability, which strategy's selection should be most trusted?

  1. Strategy 1 because p-values provide direct statistical evidence of variable importance
  2. Strategy 2 because AIC balances model fit with complexity more objectively than p-values
  3. Strategy 3 because cross-validation directly estimates out-of-sample prediction performance (correct answer)
  4. All strategies should produce similar results if applied correctly to the same dataset
  5. The strategy selecting the most variables should be trusted for comprehensive modeling
Explanation: When evaluating model selection strategies, the key question is which method best predicts how well your model will perform on new, unseen data – what we call generalizability or external validity. Cross-validation directly addresses this concern by simulating the prediction process. It repeatedly splits your data into training and testing portions, builds models on the training data, and evaluates performance on the held-out testing data. This mimics what happens when you apply your final model to future patients or samples, making it the gold standard for assessing generalizability. Strategy 3's selection of fewer variables (5) likely reflects this rigorous approach to avoiding overfitting. Option A is problematic because p-value-based selection suffers from multiple testing issues and doesn't directly measure prediction performance. Variables can be statistically significant yet contribute little to prediction accuracy, or vice versa. Option B, while AIC does balance fit and complexity better than p-values, still optimizes for in-sample performance with a penalty term. It's an improvement over p-values but doesn't directly estimate out-of-sample performance like cross-validation does. Option D is incorrect because these methods optimize different objectives and will typically yield different results. P-values focus on statistical significance, AIC on penalized likelihood, and cross-validation on prediction accuracy. Study tip: When you see model selection questions, remember that cross-validation is your best friend for generalizability. Methods that directly estimate out-of-sample performance will generally outperform those that rely on in-sample statistics with penalties.

Question 18

In a multiple regression analysis, a researcher observes that adding interaction terms between continuous variables dramatically improves model fit (R² increases from 0.42 to 0.78), but the individual interaction terms have p-values ranging from 0.03 to 0.15. What should be the primary consideration for model selection?

  1. Include all interaction terms since the overall R² improvement is substantial
  2. Include only interaction terms with p < 0.05 to maintain statistical rigor
  3. Evaluate whether the improvement represents genuine relationships or overfitting through validation (correct answer)
  4. Transform the continuous variables to eliminate the need for interaction terms
  5. Use stepwise regression to automatically select the most appropriate interaction terms
Explanation: When you encounter dramatic improvements in model fit accompanied by mixed statistical significance, you're facing a classic model selection dilemma that requires balancing explanatory power against model reliability. The substantial jump in R² from 0.42 to 0.78 suggests the interaction terms capture important relationships, but the mixed p-values (ranging from 0.03 to 0.15) create uncertainty about which effects are genuine. The primary concern here is overfitting – when a model performs well on your current data but fails to generalize to new observations. Answer C correctly identifies that validation techniques (like cross-validation or holdout samples) are essential to determine whether this improvement represents real patterns or just noise-fitting. Answer A is problematic because a high R² alone doesn't guarantee a good model – you can always improve R² by adding variables, even random ones. Answer B applies an overly rigid statistical rule without considering the bigger picture; some interaction terms with p-values slightly above 0.05 might still contribute meaningfully to model performance. Answer D suggests transforming variables to avoid interactions entirely, but this doesn't address the core question and might eliminate genuine interaction effects that provide valuable insights. Study tip: In model selection questions, remember that statistical significance and practical utility don't always align perfectly. When you see large improvements in fit statistics paired with borderline significance tests, think "validation" – the gold standard is how well your model performs on fresh data, not just the data used to build it.

Question 19

A researcher performs backward elimination starting with 20 variables, removing variables with p > 0.10 at each step. The final model contains 6 variables, all with p < 0.05, and shows good fit to the training data. When tested on new data, prediction accuracy is poor. What does this suggest about the variable selection process?

  1. The p-value threshold of 0.10 was too liberal for effective variable selection
  2. Backward elimination inherently produces less stable models than forward selection
  3. The selection process likely capitalized on sample-specific relationships that don't generalize (correct answer)
  4. The final model needs additional interaction terms to improve external validity
  5. Starting with 20 variables was insufficient to capture the true underlying relationships
Explanation: When you encounter questions about variable selection and poor external validation despite good model fit, think about overfitting and the multiple testing problem. This scenario describes a classic case where aggressive variable selection leads to models that don't generalize. The correct answer is C because backward elimination with 20 starting variables creates numerous statistical tests, dramatically increasing the chance of finding spurious relationships. Even with a 0.10 threshold, when you test many variables, some will appear significant purely by chance in your specific sample. The fact that the model fits training data well but fails on new data is the hallmark of overfitting—the selection process "memorized" noise rather than true signal. Option A incorrectly suggests the threshold was too liberal, but even stricter thresholds don't solve the fundamental multiple testing problem when starting with many variables. Option B makes an unfounded claim about backward versus forward selection stability—both methods suffer from similar overfitting issues when applied to high-dimensional data. Option D misses the point entirely by suggesting the model needs complexity (interaction terms) when the real problem is that it's already too complex and unstable. The poor external validation despite statistical significance in all retained variables is your key clue that the selection process found sample-specific patterns that don't represent true population relationships. Study tip: Remember that statistical significance ≠ practical significance or generalizability. When you see good training performance but poor test performance in variable selection contexts, always consider overfitting as the primary culprit.

Question 20

When comparing two regression models using the same dataset, Model A has an AIC of 245.7 and Model B has an AIC of 251.2. However, when both models are applied to a new validation dataset, Model A's prediction error is substantially higher than Model B's. What does this scenario most likely indicate?

  1. The AIC calculation was performed incorrectly for one of the models
  2. Model A was overfitted to the training data despite having better AIC (correct answer)
  3. Model B contains systematic bias that affects its AIC but not predictive ability
  4. The validation dataset is not representative of the original population studied
  5. AIC is an unreliable criterion for model selection in regression analysis
Explanation: When you encounter questions comparing model performance metrics versus actual predictive performance, you're dealing with the fundamental challenge of model selection and validation in biostatistics. Model A has a lower AIC (245.7 vs 251.2), suggesting it fits the training data better, since AIC penalizes models for complexity while rewarding goodness of fit. However, when tested on new data, Model A performs worse than Model B. This classic pattern indicates overfitting - Model A learned the specific patterns and noise in the training data too well, making it less generalizable to new datasets. The correct answer is B. Let's examine why the other options don't fit: A) suggests computational error, but both AIC values are plausible and the systematic nature of this problem points to a conceptual issue, not calculation mistakes. C) proposes that Model B has bias affecting AIC but not prediction - this is backwards logic since we'd expect bias to hurt predictive performance more than AIC. D) suggests the validation dataset is unrepresentative, but if this were true, we'd typically see both models performing poorly or unpredictably, not a consistent reversal where the simpler model (higher AIC) performs better. Key takeaway: AIC is excellent for comparing models on the same dataset, but lower AIC doesn't guarantee better generalization. Always validate model performance on independent data. When AIC and validation performance disagree, suspect overfitting in the model with better AIC but worse validation performance.