Ridge & Lasso Regularization — Regularization: ridge and lasso concepts (intro) - Statistics Graduate Level

SECTION 1

Historical Context & Motivation

The story of regularization in statistical estimation is fundamentally a story about the tension between fitting data well and building models that generalize reliably to unseen observations. Ordinary least squares (OLS) regression, while possessing elegant theoretical properties under classical assumptions, becomes unstable or entirely unusable when predictors are highly correlated or when the number of features rivals or exceeds the sample size. In such settings, the OLS estimator exhibits high variance — small perturbations in the training data produce wildly different coefficient estimates, a hallmark of overfitting. Regularization methods address this pathology by introducing a controlled amount of bias into the estimation process, thereby achieving a more favorable position on the bias–variance tradeoff curve.

The intellectual roots of penalized estimation extend back to the mid-twentieth century, drawing on ideas from numerical analysis, Bayesian inference, and applied statistics. Two penalization strategies have risen to particular prominence: ridge regression (ℓ₂ penalty), which shrinks all coefficients toward zero while retaining every predictor in the model, and lasso regression (ℓ₁ penalty), which can drive some coefficients to exactly zero, performing automatic variable selection. Understanding the historical trajectory of these methods illuminates why they remain indispensable tools in modern high-dimensional statistics.

1943 • Tikhonov Regularization

Andrey Tikhonov proposes adding a squared-norm penalty to stabilize ill-posed integral equations in functional analysis, providing the mathematical foundation that would later be adapted for regression.

1970 • Hoerl & Kennard — Ridge Regression

Arthur Hoerl and Robert Kennard formally introduce ridge regression to the statistics literature, demonstrating that adding λI to X^TX stabilizes coefficient estimates under multicollinearity.

1996 • Tibshirani — The Lasso

Robert Tibshirani publishes the Least Absolute Shrinkage and Selection Operator (LASSO), showing that an ℓ₁ penalty simultaneously performs shrinkage and variable selection, producing sparse solutions.

2005 • Elastic Net & Modern Extensions

Zou and Hastie propose the elastic net, combining ℓ₁ and ℓ₂ penalties to overcome limitations of lasso in the presence of grouped correlated predictors, catalyzing a wave of penalized regression research.

2010s • High-Dimensional Theory Matures

Theoretical guarantees for lasso (oracle inequalities, restricted isometry conditions) and computational advances (coordinate descent, LARS) make regularized regression standard in genomics, economics, and machine learning.

The central question that motivated these developments remains profoundly relevant: when a model has many parameters relative to the available data, how should one constrain the estimation process to obtain predictions that are both accurate and interpretable? Ridge and lasso regularization offer two complementary philosophical answers to this question, and understanding their geometric and algebraic properties is essential for any practitioner working with modern statistical or machine learning models.

SECTION 2

Core Principles & Definitions

At its heart, regularization modifies the standard OLS objective by appending a penalty term that discourages large coefficient values. The general penalized regression objective can be written as the sum of the residual sum of squares and a penalty function of the coefficient vector, scaled by a non-negative tuning parameter λ. The choice of penalty function — whether it penalizes the squared magnitude of coefficients (ℓ₂) or their absolute magnitude (ℓ₁) — determines whether the method is ridge or lasso, and this choice has profound implications for the geometry of the solution and the properties of the resulting estimator.

Bias–Variance Tradeoff

Regularization deliberately introduces bias into the estimator to achieve a larger reduction in variance. The net effect is lower mean squared error (MSE), especially when OLS would overfit.

Shrinkage toward Zero

Both ridge and lasso pull coefficient estimates toward zero. Ridge performs proportional shrinkage — all coefficients shrink but remain nonzero. Lasso can shrink coefficients all the way to exactly zero, performing variable selection.

Tuning Parameter λ

The scalar λ ≥ 0 controls the strength of regularization. When λ = 0, the solution reduces to OLS; as λ → ∞, coefficients shrink toward zero. The optimal λ is typically chosen via cross-validation.

Sparsity vs. Dense Solutions

Ridge always produces dense coefficient vectors (all nonzero). Lasso produces sparse vectors, making it ideal when the true model involves only a subset of available predictors.

Bayesian Interpretation

Ridge corresponds to a Gaussian prior on β, while lasso corresponds to a Laplace (double-exponential) prior. The penalty strength λ is inversely related to the prior variance.

✦

KEY TAKEAWAY

Think of regularization as a budget constraint on model complexity. Imagine you are furnishing a room with a fixed budget: OLS lets you buy every item (coefficient) you want regardless of cost, often resulting in a cluttered, impractical space. Ridge gives you the same budget for everything but forces you to buy cheaper versions of each item — every piece stays, but at reduced scale. Lasso forces you to choose: you can only afford a few items, so you pick the most important ones and leave the rest out entirely. The tuning parameter λ determines how tight that budget is.

SECTION 3

Geometric Interpretation

The most illuminating way to understand the fundamental difference between ridge and lasso is through their constraint region geometry. The penalized objective can equivalently be expressed as a constrained optimization problem: minimize the RSS subject to a norm constraint on the coefficient vector. For ridge, this constraint region is a circle (ℓ₂ ball) in two dimensions, while for lasso it is a diamond (ℓ₁ ball). The shape of the constraint region determines where the elliptical contours of the RSS are most likely to make first contact, and this contact point is the regularized solution.

Left: the ridge ℓ₂ constraint region is a circle; the RSS contour ellipses generically touch the smooth boundary at a point where both coefficients are nonzero. Right: the lasso ℓ₁ constraint region is a diamond with corners on the axes; contours are more likely to make first contact at a corner, setting one or more coefficients exactly to zero.

The geometry reveals the essential distinction. Because the ℓ₁ ball has sharp corners aligned with the coordinate axes, it is far more probable — especially in higher dimensions — that the expanding RSS contour ellipsoid will first touch the constraint region at a vertex where one or more coordinates equal zero. In contrast, the smooth curvature of the ℓ₂ ball means the tangent point almost surely lies in the interior of the coordinate-positive orthant, yielding nonzero estimates for every coefficient. This geometric argument extends naturally to p dimensions: the ℓ₁ ball in ℝᵖ is a cross-polytope whose vertices lie on axes, while the ℓ₂ ball is a hypersphere with no such preferred directions.

SECTION 4

Mathematical Framework

We now formalize the penalized regression objectives. Consider the standard linear model y = Xβ + ε, where X is an n × p design matrix (assumed centered and possibly standardized), y is an n × 1 response vector, β is a p × 1 coefficient vector, and ε ~ N(0, σ²Iₙ). The OLS estimator minimizes ‖y − Xβ‖₂². Regularized estimators augment this with a penalty on β.

RIDGE REGRESSION OBJECTIVE

β̂_ridge = argmin_β { ‖y − Xβ‖₂² + λ‖β‖₂² }

Here ‖β‖₂² = Σⱼ βⱼ² is the squared ℓ₂ norm. The closed-form solution is β̂_ridge = (X^TX + λI)⁻¹X^Ty. The addition of λI to X^TX ensures invertibility even when X^TX is singular.

LASSO REGRESSION OBJECTIVE

β̂_lasso = argmin_β { ‖y − Xβ‖₂² + λ‖β‖₁ }

Here ‖β‖₁ = Σⱼ |βⱼ| is the ℓ₁ norm. There is no closed-form solution; the objective is convex but not differentiable at βⱼ = 0. Solutions are obtained via coordinate descent, LARS, or subgradient methods.

SVD Perspective on Ridge Shrinkage

The singular value decomposition (SVD) X = UDV^T provides deep insight into ridge shrinkage. In terms of the SVD, the OLS fitted values are ŷ_OLS = Σⱼ uⱼ(dⱼ²/dⱼ²) uⱼ^Ty, while the ridge fitted values are ŷ_ridge = Σⱼ uⱼ(dⱼ²/(dⱼ² + λ)) uⱼ^Ty. The factor dⱼ²/(dⱼ² + λ) is a shrinkage factor that is close to 1 for large singular values (directions of high variance in X) and close to 0 for small singular values (noisy directions). Ridge thus shrinks the most precisely those components of the solution that are estimated with the least precision.

RIDGE SHRINKAGE FACTORS (SVD FORM)

β̂_ridge = V diag(d₁²/(d₁²+λ), …, dₚ²/(dₚ²+λ)) V^T β̂_OLS

Each OLS coefficient in the rotated coordinate system (principal component space) is multiplied by the shrinkage factor dⱼ²/(dⱼ² + λ), which lies strictly between 0 and 1 for λ > 0.

LASSO SOFT-THRESHOLDING (UNIVARIATE CASE)

β̂_lasso,j = sign(β̂_OLS,j) × max(|β̂_OLS,j| − λ/2, 0)

In the orthonormal design case (X^TX = I), the lasso solution is the soft-thresholding operator applied to each OLS coefficient independently. Coefficients with |β̂_OLS,j| < λ/2 are set to exactly zero.

SECTION 5

Coefficient Shrinkage Paths & Comparison

A powerful way to visualize the behavior of ridge and lasso is to plot the coefficient path — the trajectory of each βⱼ estimate as λ varies from 0 (OLS solution) to ∞ (all coefficients at zero). For ridge, these paths are smooth curves that decay asymptotically toward zero but never reach it. For lasso, the paths are piecewise linear, and each coefficient hits exactly zero at a finite value of λ. The diagram below illustrates representative coefficient paths for a three-predictor model under both methods.

Coefficient paths for three predictors as λ increases. Ridge (left): all paths smoothly approach but never reach the zero line. Lasso (right): paths are piecewise linear and hit zero at specific λ values (marked by dots), with β₂ being eliminated first.

Property	Ridge (ℓ₂)	Lasso (ℓ₁)
Penalty term	λ Σ βⱼ²	λ Σ \|βⱼ\|
Constraint geometry	Hypersphere (smooth)	Cross-polytope (corners)
Closed-form solution	Yes: (X^TX + λI)⁻¹X^Ty	No (requires iterative algorithms)
Sparsity	Never — all coefficients remain nonzero	Yes — some coefficients set to exactly zero
Correlated predictors	Shrinks correlated predictors together	Tends to select one and discard others
Bayesian prior	Gaussian: βⱼ ~ N(0, σ²/λ)	Laplace: βⱼ ~ Laplace(0, 1/λ)
Coefficient path	Smooth, asymptotic to zero	Piecewise linear, reaches zero at finite λ

Key differences between ridge and lasso regularization

SECTION 6

Worked Example: Ridge and Lasso on a Simple Dataset

Consider a simple regression problem with n = 4 observations and p = 2 standardized predictors. Suppose that X^TX = [[5, 4], [4, 5]] and X^Ty = [7, 3]. We will compute both the OLS and ridge estimates with λ = 2, then discuss the lasso solution qualitatively.

Ridge Regression with λ = 2

1
Step 1 — Compute OLS Estimate

The OLS solution solves (X^TX)β = X^Ty. We compute (X^TX)⁻¹ = (1/(25−16))[[5,−4],[−4,5]] = (1/9)[[5,−4],[−4,5]]. Then β̂_OLS = (1/9)[[5,−4],[−4,5]] × [7,3] = (1/9)[35−12, −28+15] = (1/9)[23, −13].

β̂_OLS = (2.556, −1.444)
2
Step 2 — Form Penalized Matrix ( $X^TX$ + λI)

For λ = 2, add 2I to X^TX: [[5+2, 4],[4, 5+2]] = [[7, 4],[4, 7]]. The determinant is 49 − 16 = 33.

X^TX + λI = [[7, 4], [4, 7]], det = 33
3
Step 3 — Compute Ridge Estimate

Invert and multiply: β̂_ridge = (1/33)[[7,−4],[−4,7]] × [7,3] = (1/33)[49−12, −28+21] = (1/33)[37, −7].

β̂_ridge = (1.121, −0.212)
4
Step 4 — Analyze Shrinkage

Comparing estimates: β₁ shrank from 2.556 to 1.121 (56% reduction), and β₂ shrank from −1.444 to −0.212 (85% reduction). Both coefficients moved substantially toward zero, with the smaller coefficient (β₂) experiencing proportionally greater shrinkage. Crucially, neither coefficient is exactly zero — this is characteristic of ridge.

Shrinkage: β₁ reduced 56%, β₂ reduced 85% — both nonzero
5
Step 5 — Lasso Comparison (Qualitative)

For lasso with a comparable penalty, the soft-thresholding nature of ℓ₁ would likely set β₂ to exactly zero (since its OLS magnitude is smaller), leaving only β₁ in the model. Numerical coordinate descent would yield approximately β̂_lasso ≈ (1.5, 0) for an appropriately chosen λ, producing a genuinely sparse model.

Lasso: β̂ ≈ (1.5, 0) — β₂ eliminated

SECTION 7

Strengths, Limitations, and When to Use Each

Neither ridge nor lasso is universally superior; each excels in different data environments. Choosing between them — or combining them via the elastic net — depends on the underlying structure of the data-generating process, the goals of the analysis (prediction vs. interpretation), and practical considerations such as computational cost and the number of predictors relative to observations.

Criterion	Ridge Strengths / Limitations	Lasso Strengths / Limitations
When many predictors are relevant	Strength: retains all predictors, distributes shrinkage evenly	Limitation: may discard useful predictors arbitrarily
When true model is sparse	Limitation: cannot zero out irrelevant predictors	Strength: automatic variable selection produces interpretable models
Multicollinearity	Strength: handles correlated predictors gracefully by shrinking together	Limitation: arbitrarily selects one predictor from correlated group
p > n settings	Applicable; solution always exists and is unique	Applicable; selects at most n predictors
Computation	Efficient: closed-form matrix solution	Requires iterative algorithms (coordinate descent)
Interpretability	All predictors present; harder to interpret	Sparse models; easier to communicate results

Comparative strengths and limitations of ridge versus lasso regularization

⚖

WHEN TO USE WHICH

A useful heuristic: if you believe that most predictors contribute at least some signal (e.g., each gene in a microarray has a small additive effect), prefer ridge. If you believe that only a handful of predictors drive the response (e.g., a few economic indicators predict GDP growth), prefer lasso. When you are unsure, the elastic net (combining both penalties) provides a robust middle ground and lets cross-validation decide the mixing proportion.

SECTION 8

Connections to Advanced Theory

Ridge and lasso are foundational members of a larger family of penalized estimators that have become central to modern high-dimensional statistics. Understanding their theoretical properties opens the door to more advanced topics including the elastic net, group lasso, adaptive lasso, and SCAD/MCP (non-convex) penalties. The theoretical landscape also encompasses oracle properties, restricted eigenvalue conditions, and connections to compressed sensing.

Concept	Ridge / Lasso (This Lesson)	Advanced Extension
Penalty function	ℓ₁ or ℓ₂ penalty separately	Elastic net: α‖β‖₁ + (1−α)‖β‖₂², bridging ridge and lasso
Variable selection consistency	Standard lasso: not always oracle consistent	Adaptive lasso with data-driven weights achieves oracle property
Grouped predictors	Lasso selects one from a correlated group	Group lasso enforces all-in or all-out selection within groups
Bias properties	Both introduce non-negligible bias for large signals	SCAD/MCP: non-convex penalties that reduce bias for large coefficients
Bayesian framework	Gaussian or Laplace priors on β	Spike-and-slab priors, horseshoe priors for more principled shrinkage

From introductory to advanced regularization concepts

💡

Looking Ahead

The introductory concepts of ℓ₁ and ℓ₂ penalization covered here form the bedrock for graduate-level topics including high-dimensional inference (de-biased lasso for confidence intervals), penalized likelihood methods for GLMs, and matrix regularization in multivariate and tensor regression. Mastering the geometry and algebraic structure of ridge and lasso is prerequisite to understanding these extensions.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

Explain geometrically why the lasso penalty (ℓ₁ norm) can produce exact zeros in the coefficient estimates while the ridge penalty (ℓ₂ norm) cannot. Your answer should reference the shapes of the respective constraint regions and how RSS contour ellipsoids interact with them.

Show answer

The ℓ₁ constraint region in ℝᵖ is a cross-polytope (diamond) whose vertices lie on the coordinate axes. The ℓ₂ constraint region is a smooth hypersphere with no corners. When we expand the RSS contour ellipsoids outward from the OLS solution, the first point of tangency with the constraint region is the regularized estimate. Because the ℓ₁ ball has sharp corners on the coordinate axes, the contour ellipsoid has a high probability of first touching the constraint region at one of these corners, where one or more coordinates equal zero. The smooth ℓ₂ ball, by contrast, has no preferred directions, so the tangent point generically has all coordinates nonzero. In higher dimensions, the proportion of the ℓ₁ ball's surface area concentrated near corners increases, making sparsity even more likely.

Question 2: BASIC CALCULATION

Suppose X^TX = I₂ (the 2×2 identity matrix) and X^Ty = [4, 1]^T. Compute the ridge estimate β̂_ridge for λ = 1. Then compute the lasso estimate using the soft-thresholding formula with λ = 1 (note: the threshold is λ/2 = 0.5 in this formulation).

Show answer

For ridge with X^TX = I: β̂_ridge = (I + λI)⁻¹X^Ty = (1/(1+1))[4, 1]^T = [2.0, 0.5]^T. Both coefficients are shrunk by the factor 1/(1+λ) = 0.5 but remain nonzero. For lasso with orthonormal design, β̂_lasso,j = sign(β̂_OLS,j) × max(|β̂_OLS,j| − λ/2, 0). With β̂_OLS = [4, 1] and λ/2 = 0.5: β̂_lasso,1 = sign(4) × max(4 − 0.5, 0) = 3.5 and β̂_lasso,2 = sign(1) × max(1 − 0.5, 0) = 0.5. Here both remain nonzero because both |β̂_OLS,j| > 0.5. If λ were increased to 2 (threshold = 1), β₂ would be set to exactly zero.

Question 3: INTERMEDIATE

Consider the SVD-based expression for ridge: β̂_ridge = V diag(dⱼ²/(dⱼ² + λ)) V^T β̂_OLS. Suppose the singular values of X are d₁ = 10, d₂ = 1, d₃ = 0.1, and λ = 1. Compute the shrinkage factor for each singular value. What does this tell you about how ridge treats directions of high versus low variance in the predictor space?

Show answer

The shrinkage factors are: d₁²/(d₁² + λ) = 100/101 ≈ 0.990 (almost no shrinkage), d₂²/(d₂² + λ) = 1/2 = 0.500 (moderate shrinkage), d₃²/(d₃² + λ) = 0.01/1.01 ≈ 0.0099 (near-total shrinkage). Ridge selectively attenuates components of the coefficient vector that correspond to directions of low variance in X (small singular values), where OLS estimates are most unstable. The component along d₃ — a direction in which the data provide almost no information — is shrunk by over 99%. The component along d₁ — a well-estimated direction — is essentially untouched. This selective shrinkage is the mechanism by which ridge achieves variance reduction with minimal increase in bias for well-determined components.

Question 4: APPLIED

A researcher has a dataset with n = 100 observations and p = 80 gene expression features to predict a continuous health outcome. Preliminary OLS regression shows high variance in coefficient estimates and several predictors with large coefficients of opposite signs. The researcher suspects that perhaps 10–15 genes are truly relevant. Should the researcher use ridge, lasso, or elastic net? Justify your recommendation, addressing the issues of multicollinearity, sparsity, and the p/n ratio.

Show answer

Given the researcher's belief that only 10–15 of 80 predictors are truly relevant, a sparse method is appropriate, ruling out pure ridge (which would retain all 80 predictors). However, gene expression data typically exhibits high correlations among genes in the same pathway, and lasso is known to behave erratically with correlated predictors — it tends to select one gene from a correlated group arbitrarily and discard the others, yielding unstable selections across bootstrap samples. The elastic net is the recommended choice: its ℓ₁ component induces sparsity consistent with the researcher's prior belief about model structure, while its ℓ₂ component handles multicollinearity by encouraging correlated genes to receive similar (nonzero or zero) coefficients. With p/n = 0.8, the problem is near-high-dimensional, further supporting regularization. The mixing parameter α and overall penalty λ should be selected via k-fold cross-validation.

Question 5: CRITICAL THINKING

The standard lasso is not selection-consistent in general: it may fail to identify the true support set even as n → ∞ unless the so-called irrepresentable condition holds. Explain informally what the irrepresentable condition requires, why it might fail in practice, and describe how the adaptive lasso addresses this limitation. How does the adaptive lasso's use of data-dependent weights relate to the Bayesian interpretation of the penalty?

Show answer

The irrepresentable condition requires that the correlation between the irrelevant predictors and the relevant predictors, mediated through the design matrix, is bounded strictly below 1 in a specific matrix-norm sense. Informally, it requires that the 'noise' predictors cannot be well-represented as linear combinations of the 'signal' predictors. This condition fails when irrelevant predictors are highly correlated with relevant ones — a common scenario in genomics or economics. The adaptive lasso addresses this by using a weighted ℓ₁ penalty: λ Σⱼ wⱼ|βⱼ|, where wⱼ = 1/|β̂_init,j|^γ for some initial consistent estimator β̂_init (often ridge or OLS) and γ > 0. Large initial estimates receive small weights (light penalty), while near-zero initial estimates receive large weights (heavy penalty), making it easier to correctly zero them out. Under mild regularity conditions, the adaptive lasso achieves the oracle property: it selects the correct model and estimates nonzero coefficients at the parametric √n rate. From a Bayesian viewpoint, the adaptive weights correspond to predictor-specific priors with different scales — coefficients believed a priori to be large receive diffuse (non-informative) priors, while those believed to be near zero receive concentrated priors strongly favoring zero.

SUMMARY

Summary & Review

Summary

Ridge regression and lasso regression are penalized estimation methods that modify the OLS objective by adding a constraint on the coefficient vector. Ridge uses an ℓ₂ (squared norm) penalty, which shrinks all coefficients proportionally toward zero while retaining every predictor, yielding a closed-form solution via ( $X^TX$ + λI)⁻¹X^Ty. Lasso uses an ℓ₁ (absolute value) penalty, whose diamond-shaped constraint region causes the solution to lie at corners of the polytope, setting some coefficients to exactly zero and thereby performing automatic variable selection.

The tuning parameter λ controls the strength of regularization, interpolating between OLS (λ = 0) and full shrinkage (λ → ∞), and is typically selected via cross-validation. From a Bayesian perspective, ridge corresponds to a Gaussian prior on β and lasso to a Laplace prior. Ridge excels when many predictors carry small signals and multicollinearity is present; lasso excels when the true model is sparse. The elastic net combines both penalties for situations where neither pure approach is optimal, and extensions such as the adaptive lasso achieve oracle consistency under weaker conditions than the standard lasso.