Model Selection & Overfitting — Model selection and overfitting concepts (intro) - Biostatistics

SECTION 1

Historical Context & Motivation

Statistical modeling has always faced a fundamental tension: a model must be complex enough to capture genuine patterns in data, yet simple enough to generalize beyond the observations used to build it. In the biomedical sciences, where researchers routinely fit regression models to predict clinical outcomes—disease risk, drug response, patient survival—getting this balance wrong can have serious consequences. An overfitted model that memorizes quirks of one patient cohort may fail catastrophically when applied to a new population, producing misleading predictions that erode clinical trust.

The intellectual roots of model selection stretch back to debates about scientific parsimony that predate modern statistics. The philosophical principle known as Occam's razor—entities should not be multiplied beyond necessity—guided early scientists to prefer simpler explanations. As regression methods matured through the twentieth century, statisticians translated this intuition into formal criteria that could quantify the trade-off between model fit and model complexity. These criteria remain central to biostatistical practice today.

1805 • Least Squares Introduced

Adrien-Marie Legendre publishes the method of least squares, establishing the foundational framework for fitting models to observed data and minimizing residual error.

1929 • Fisher & Degrees of Freedom

R. A. Fisher formalizes the concept of degrees of freedom, providing an early mechanism to penalize model complexity in the analysis of variance (ANOVA) framework.

1973 • Akaike Information Criterion (AIC)

Hirotugu Akaike proposes AIC, linking information theory to model selection and giving researchers a principled way to compare non-nested models while penalizing excess parameters.

1978 • Schwarz / Bayesian Information Criterion (BIC)

Gideon Schwarz derives BIC from a Bayesian perspective, introducing a stronger penalty for model complexity that favors more parsimonious models as sample size grows.

1995 • Cross-Validation in Biostatistics

The widespread adoption of k-fold cross-validation and bootstrapping in clinical prediction modeling, championed by Frank Harrell and others, brings resampling-based model evaluation into routine biostatistical practice.

The central question that this lesson addresses is deceptively simple: given a set of candidate models for a biomedical outcome, how do we choose the one that will perform best on future, unseen data? Answering this question requires understanding what overfitting is, why it arises, and what tools—information criteria, cross-validation, regularization—exist to guard against it.

SECTION 2

Core Principles & Definitions

Before diving into formal criteria, it is essential to establish the conceptual vocabulary that underpins the entire model-selection enterprise. Every dataset contains two components: a systematic signal (the true relationship between predictors and outcome) and random noise (measurement error, biological variability, unmeasured confounders). A model's task is to capture as much signal as possible while ignoring noise. When a model begins fitting noise, it is said to overfit the training data.

Bias

The error introduced when a model is too simple to capture the true underlying relationship. A linear regression fit to a quadratic trend will systematically miss the curvature, producing high bias (underfitting).

Variance

The sensitivity of the fitted model to the particular training sample. Highly flexible models can change dramatically from one dataset to the next, producing high variance (overfitting).

Bias–Variance Trade-off

Reducing bias typically increases variance and vice versa. Model selection seeks the complexity level—the sweet spot—where the sum of squared bias and variance (i.e., expected prediction error) is minimized.

Training vs. Test Error

Training error always decreases (or stays the same) as model complexity grows. Test error on unseen data initially decreases but eventually rises—the hallmark U-shaped curve of overfitting.

Parsimony

The principle that among models with similar predictive performance, the simplest should be preferred. Parsimonious models are easier to interpret, more robust, and less prone to overfitting—critical qualities in clinical decision-making.

✦

KEY TAKEAWAY

Think of model selection like tuning a radio. If the dial is set too low (underfitting), you hear mostly static because the receiver is not sensitive enough to pick up the station. If you crank it too high (overfitting), you amplify background interference along with the music. The goal is to find the precise dial position where the signal is clearest and the noise is minimal—the optimal bias–variance trade-off.

SECTION 3

The Bias–Variance Trade-off Visualized

The most iconic visualization in model selection is the U-shaped curve that plots prediction error against model complexity. As complexity increases from left to right, training error falls monotonically—the model simply gets better at fitting the data it has already seen. However, test error follows a U shape: it first drops (as the model captures real signal) and then climbs (as the model begins chasing noise). The optimal model complexity lies at the minimum of the test-error curve.

The training error (cyan) decreases steadily with complexity. The test error (pink) traces a U shape; its minimum at the green dashed line marks the optimal complexity. The decomposition into bias² (violet, dashed) and variance (amber, dashed) shows their opposing trends.

Notice how the gap between training error and test error widens dramatically in the overfitting region on the right side of the diagram. This gap is itself a diagnostic: if a model performs extremely well on training data but poorly on held-out data, overfitting is almost certainly at play. In biostatistics, where sample sizes are often modest relative to the number of candidate predictors (genomic covariates, imaging features, laboratory assays), this gap can appear earlier and more sharply than in data-rich domains.

SECTION 4

Mathematical Framework

The bias–variance decomposition provides the theoretical backbone for understanding overfitting. Consider a regression setting where we wish to predict an outcome Y from predictors X. Suppose the true data-generating process is Y = f(X) + ε, where ε is irreducible noise with E[ε] = 0 and Var(ε) = σ². For an estimated model f̂(X) trained on a particular sample, the expected prediction error (EPE) at a point x₀ can be decomposed as follows.

BIAS–VARIANCE DECOMPOSITION

EPE(x₀) = [Bias(f̂(x₀))]² + Var(f̂(x₀)) + σ²

Where Bias(f̂(x₀)) = E[f̂(x₀)] − f(x₀), measuring systematic error; Var(f̂(x₀)) captures the variability of f̂ across different training samples; and σ² is the irreducible error that no model can eliminate.

Two widely used information criteria translate this decomposition into practical tools for model comparison. Both start with the model's maximized log-likelihood and add a penalty proportional to the number of estimated parameters.

AKAIKE INFORMATION CRITERION (AIC)

AIC = −2 × ln(L̂) + 2k

Where L̂ is the maximized likelihood of the model and k is the number of estimated parameters. Lower AIC indicates a better model. The 2k penalty discourages unnecessary parameters.

BAYESIAN INFORMATION CRITERION (BIC)

BIC = −2 × ln(L̂) + k × ln(n)

Where n is the sample size. Because ln(n) > 2 for n ≥ 8, BIC penalizes complexity more heavily than AIC, favoring simpler models, especially in large samples.

ADJUSTED R-SQUARED

R²_adj = 1 − [(1 − R²) × (n − 1) / (n − k − 1)]

A modification of R² that penalizes the addition of predictors that do not improve the model sufficiently. Unlike ordinary R², R²_adj can decrease when a newly added predictor contributes more noise than signal.

⚠

Why Penalize Complexity?

Without a complexity penalty, the model with the most parameters will always fit the training data best—even if many parameters merely capture noise. Information criteria formalize the intuition that each additional parameter must 'earn its keep' by improving fit enough to offset the risk of overfitting. In biostatistical applications, a common guideline is to include no more than about n/10 to n/15 candidate predictors (where n is the number of outcome events) to avoid unstable coefficient estimates.

SECTION 5

Model Validation Strategies

Information criteria provide a quick, computation-free approximation to out-of-sample performance, but cross-validation offers a more direct estimate by actually evaluating models on data that were not used for fitting. In biostatistics, cross-validation has become a gold-standard tool for assessing predictive accuracy, especially when the goal is to deploy a model in clinical practice.

In 5-fold cross-validation, the dataset is split into five equally sized folds. Each fold takes a turn serving as the test set (pink) while the remaining four folds (cyan) form the training set. The model's prediction error is averaged across all five iterations to produce a robust estimate of out-of-sample performance.

Several variants of cross-validation exist. Leave-one-out cross-validation (LOOCV) sets K = n, using each observation as its own test set. This gives a nearly unbiased estimate of prediction error but is computationally expensive and can have high variance. In practice, 5- or 10-fold CV provides a good compromise. For time-to-event outcomes common in clinical research (e.g., survival analysis), specialized methods such as optimism-corrected bootstrap validation are often preferred because they account for the censoring structure of the data. The key principle underlying all these techniques is the same: never evaluate a model solely on the data used to train it.

Validation Method	Approach	Pros	Cons
Train/Test Split	Hold out a fixed fraction (e.g., 30%) as a test set	Simple, fast, easy to implement	High variance; result depends on the particular split
K-Fold CV	Rotate test fold through K partitions of data	Lower variance than single split; efficient use of data	K models must be fitted; slight bias for small K
LOOCV	Each observation is its own test set (K = n)	Nearly unbiased error estimate	Very high computational cost; high estimate variance
Bootstrap	Resample with replacement; evaluate on out-of-bag cases	Low variance; handles complex sampling designs	Can be optimistically biased without correction (e.g., 0.632+ method)

Comparison of common model validation strategies

SECTION 6

Worked Example: Selecting a Logistic Regression Model

Suppose a biostatistician has collected data on n = 200 patients with a binary outcome (disease present vs. absent). Three candidate logistic regression models are under consideration, each with different sets of predictors. We will compare them using AIC and BIC to illustrate the model-selection process.

Comparing Three Candidate Models with AIC and BIC

1
Step 1 — Define the Candidate Models

Model A (simple): 2 predictors (age, BMI) → k = 3 parameters (2 slopes + intercept). Model B (moderate): 5 predictors (age, BMI, blood pressure, cholesterol, smoking status) → k = 6. Model C (complex): 12 predictors (including interaction terms and polynomial terms) → k = 13.
2
Step 2 — Obtain Maximized Log-Likelihoods

After fitting each model via maximum likelihood estimation, we obtain: ln(L̂_A) = −118.5, ln(L̂_B) = −105.2, ln(L̂_C) = −100.8. As expected, the more complex model achieves a higher (less negative) log-likelihood because it fits the training data more closely.
3
Step 3 — Compute AIC for Each Model

Using AIC = −2 × ln(L̂) + 2k: AIC_A = −2(−118.5) + 2(3) = 237.0 + 6.0 = 243.0. AIC_B = −2(−105.2) + 2(6) = 210.4 + 12.0 = 222.4. AIC_C = −2(−100.8) + 2(13) = 201.6 + 26.0 = 227.6.

By AIC, Model B is preferred (AIC = 222.4, lowest).
4
Step 4 — Compute BIC for Each Model

Using BIC = −2 × ln(L̂) + k × ln(n), with ln(200) ≈ 5.30: BIC_A = 237.0 + 3(5.30) = 237.0 + 15.9 = 252.9. BIC_B = 210.4 + 6(5.30) = 210.4 + 31.8 = 242.2. BIC_C = 201.6 + 13(5.30) = 201.6 + 68.9 = 270.5.

By BIC, Model B is again preferred (BIC = 242.2). Note how BIC penalizes Model C much more severely than AIC does.
5
Step 5 — Interpret the Results

Both AIC and BIC agree that Model B strikes the best balance between goodness of fit and complexity. Model C achieves a slightly better fit (lower −2 ln L̂) but its 13 parameters are not sufficiently justified by the improvement. With only 200 observations, Model C's 13 parameters approach the guideline limit of n/15 ≈ 13.3, raising concerns about overfitting. Model B, with 6 parameters, is well within safe bounds.

Select Model B with 5 predictors as the final model.

SECTION 7

Comparing Model-Selection Criteria

No single model-selection criterion is universally optimal; each embeds different assumptions about the goal of modeling. Understanding the strengths and weaknesses of each approach helps biostatisticians choose the right tool for a given research question.

Criterion	Philosophical Basis	Penalty Strength	Best For	Limitation
AIC	Kullback–Leibler divergence minimization; asymptotically equivalent to LOOCV	Moderate (2k)	Prediction-focused modeling	May select overly complex models in small samples
BIC	Bayesian model posterior probability; consistent estimator of true model	Strong (k × ln n)	Identifying the 'true' model structure	Assumes the true model is in the candidate set
Adjusted R²	Penalized proportion of variance explained	Mild	Quick comparisons of linear regression models	Not applicable to logistic regression or survival models
Cross-Validation	Direct estimation of out-of-sample prediction error	Implicit (data-driven)	Any predictive modeling context; non-parametric models	Computationally expensive; results vary with fold assignment

Comparison of major model-selection criteria

✦

KEY TAKEAWAY

If your primary goal is prediction (e.g., building a clinical risk score to deploy at the bedside), AIC or cross-validation is typically the better guide because they optimize predictive accuracy. If your goal is inference (e.g., identifying which biological pathways truly drive a disease), BIC's stronger penalty helps exclude spurious associations. In practice, many biostatisticians report both AIC and BIC—when they agree, confidence in the selected model is strengthened.

SECTION 8

Connection to Regularization & Advanced Methods

The model-selection criteria discussed so far operate by choosing among a discrete set of candidate models. An alternative paradigm—regularization—addresses overfitting from within the estimation procedure itself. Instead of selecting or dropping predictors outright, regularized methods impose a continuous penalty on the magnitude of regression coefficients, shrinking them toward zero. This approach is particularly powerful in high-dimensional biomedical settings (e.g., genomics, proteomics) where the number of candidate predictors p may rival or exceed the sample size n.

Feature	Traditional Model Selection (AIC/BIC)	Regularization (Ridge, LASSO)
How complexity is controlled	Compare discrete candidate models; select the one with best criterion value	Penalize large coefficients continuously via a tuning parameter λ
Variable selection	Explicit: predictors are included or excluded	LASSO can shrink coefficients exactly to zero; Ridge shrinks but retains all
High-dimensional data (p >> n)	Difficult; requires pre-screening or stepwise methods	Naturally handles p >> n through shrinkage
Tuning	No tuning parameter; penalty is built into the criterion formula	λ is typically selected via cross-validation
Interpretability	Straightforward: selected model coefficients are ordinary estimates	Coefficients are biased toward zero; inference requires special methods

Traditional model selection vs. regularization approaches

In more advanced courses, you will encounter Ridge regression (L2 penalty) and LASSO (L1 penalty), as well as the elastic net (a blend of both). These methods seamlessly integrate model selection and estimation into a single step and have become indispensable tools in modern biostatistical practice, especially for building prediction models from electronic health records, imaging data, and multi-omics platforms. The concepts introduced in this lesson—bias–variance trade-off, the danger of evaluating on training data, and the need for out-of-sample validation—remain the conceptual foundation upon which all these advanced techniques are built.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

A researcher fits a 20-predictor logistic regression model to a dataset of 80 patients and reports a training accuracy of 96%. When the model is applied to a new cohort of 80 patients, accuracy drops to 61%. Explain which phenomenon is most likely responsible and identify the key evidence from the problem statement that supports your answer.

Show answer

This is a classic case of overfitting. The large gap between training accuracy (96%) and test accuracy (61%) is the primary diagnostic signal. With 20 predictors and only 80 patients, the ratio of observations to predictors (80/20 = 4) is far below the recommended guideline of 10–15 events per predictor. The model had enough flexibility to memorize idiosyncratic patterns—noise—in the training data that do not generalize. A simpler model with fewer predictors, chosen via AIC/BIC or cross-validation, would likely perform more consistently across cohorts.

Question 2: BASIC CALCULATION

Two linear regression models are fitted to the same dataset of n = 150 observations. Model 1 has k = 4 parameters and ln(L̂) = −210.3. Model 2 has k = 7 parameters and ln(L̂) = −205.8. Compute the AIC and BIC for each model and state which model is preferred under each criterion.

Show answer

For AIC = −2 × ln(L̂) + 2k: AIC₁ = −2(−210.3) + 2(4) = 420.6 + 8 = 428.6; AIC₂ = −2(−205.8) + 2(7) = 411.6 + 14 = 425.6. Model 2 is preferred by AIC (425.6 < 428.6). For BIC = −2 × ln(L̂) + k × ln(n), with ln(150) ≈ 5.01: BIC₁ = 420.6 + 4(5.01) = 420.6 + 20.04 = 440.6; BIC₂ = 411.6 + 7(5.01) = 411.6 + 35.07 = 446.7. Model 1 is preferred by BIC (440.6 < 446.7). The criteria disagree because BIC's stronger penalty (5.01 vs. 2 per parameter) outweighs the marginal improvement in fit that Model 2 provides.

Question 3: INTERMEDIATE

A biostatistician uses 10-fold cross-validation to evaluate a Cox proportional hazards model for predicting patient survival. The C-statistic (concordance index) on each fold is: 0.78, 0.74, 0.81, 0.76, 0.72, 0.79, 0.77, 0.73, 0.80, 0.75. (a) Calculate the cross-validated C-statistic. (b) If the apparent C-statistic on the full training data is 0.85, what does the discrepancy suggest? (c) Would you recommend external validation? Why?

Show answer

(a) The cross-validated C-statistic = (0.78 + 0.74 + 0.81 + 0.76 + 0.72 + 0.79 + 0.77 + 0.73 + 0.80 + 0.75) / 10 = 7.65 / 10 = 0.765. (b) The optimism = 0.85 − 0.765 = 0.085, indicating that the model's apparent discrimination overstates its true predictive ability by about 8.5 percentage points—a clear sign of moderate overfitting. (c) Yes, external validation on an independent cohort is strongly recommended. Internal validation via cross-validation estimates optimism, but it cannot detect issues such as population-specific effects, differences in measurement protocols, or temporal drift in disease prevalence. External validation is the gold standard for confirming a model's clinical utility.

Question 4: APPLIED

A clinical research team is developing a risk score for 30-day hospital readmission using electronic health record data. They have 2,000 patients with 350 readmission events. The candidate predictor pool includes 60 variables. (a) Using the events-per-variable (EPV) guideline of 10–15 events per predictor, how many predictors can be safely included? (b) Propose a model-selection strategy that respects this constraint while exploring the full predictor pool.

Show answer

(a) With 350 events, the EPV guideline allows approximately 350/15 ≈ 23 to 350/10 = 35 predictors. Including all 60 candidate predictors (EPV ≈ 5.8) would risk substantial overfitting. (b) A recommended strategy involves pre-specification based on clinical knowledge: first reduce the pool to ~25–30 predictors based on prior literature and clinical relevance, then use backward elimination guided by AIC within a bootstrap-resampling framework (e.g., Harrell's approach) to identify a stable subset. Alternatively, one could fit a LASSO-penalized logistic regression to all 60 predictors, using 10-fold cross-validation to select the tuning parameter λ, which would effectively shrink many coefficients to zero. The final model's discrimination and calibration should be assessed via internal validation (optimism-corrected bootstrap or repeated CV) and ideally external validation in a separate institution's data.

Question 5: CRITICAL THINKING

Consider this claim: 'AIC and BIC can never agree on the same model, because AIC favors complexity and BIC favors simplicity.' Is this statement correct? Provide a formal argument or counterexample. Then discuss the conditions under which AIC and BIC are most likely to disagree, referencing the structure of their penalty terms.

Show answer

The statement is incorrect. AIC and BIC can and often do agree. They share the same goodness-of-fit term (−2 × ln L̂) and differ only in the penalty: AIC uses 2k while BIC uses k × ln(n). Both penalties are proportional to k, so their rankings differ only when the improvement in −2 ln L̂ from adding a predictor falls between the two penalty increments—i.e., between 2 and ln(n) per parameter. If the improvement is either clearly above ln(n) (both select the larger model) or clearly below 2 (both select the smaller model), they agree. Disagreement is most likely when (1) the sample size is large (making ln(n) >> 2, so BIC is much stricter), and (2) the marginal improvement from additional predictors is moderate. For n ≤ 7, ln(n) < 2, so BIC is actually less penalizing than AIC—though this scenario rarely arises in biostatistical practice. This analysis underscores that the criteria embody different inferential goals rather than a simple complexity preference.

SUMMARY

Lesson Summary

Summary

Model selection in biostatistics addresses the fundamental tension between fitting data well and generalizing to new observations. The bias–variance trade-off frames this challenge mathematically: simple models suffer from high bias (underfitting), while overly complex models exhibit high variance (overfitting). The hallmark diagnostic of overfitting is a large gap between training error and test error. Information criteria such as AIC and BIC quantify this trade-off by penalizing model complexity, with BIC applying a stronger penalty that scales with sample size.

Cross-validation provides a direct, data-driven estimate of out-of-sample performance by rotating training and test partitions across the data. The events-per-variable guideline (10–15 events per predictor) offers a practical safeguard against overfitting in clinical prediction models. Looking ahead, regularization methods like LASSO and Ridge regression extend these ideas by embedding complexity control directly within the estimation process, enabling principled modeling even in high-dimensional settings. Mastering these foundational concepts equips you to build biostatistical models that are not only accurate on today's data but robust enough to guide clinical decisions for future patients.