Ridge & Lasso Regularization - Statistics Graduate Level

1. Which prior specifications produce these estimators as posterior modes with the stated values of $\lambda$?
   Answer: Ridge uses independent $N(0,\tau^2)$ priors with $\lambda=\sigma^2/\tau^2$; lasso uses independent Laplace priors of scale $b$ with $\lambda=\sigma^2/b$. When you see a question connecting regularized regression to Bayesian posteriors, the key move is to write out the log-posterior and match terms to the penalized loss. Recall that maximizing the posterior is equivalent to minimizing the negative log-posterior. With a Gaussian likelihood, $-\log p(y|\beta) \propto \frac{1}{2\sigma^2}||y - X\beta||_2^2$. The prior contributes $-\log p(\beta)$, and the penalty structure of that term determines which estimator you recover. For a Normal prior $\beta_j \sim N(0, \tau^2)$, we have $-\log p(\beta) \propto \frac{1}{2\tau^2}||\beta||_2^2$. Combined with the likelihood (scaled by $\sigma^2$), the negative log-posterior becomes $\frac{1}{2}||y-X\beta||_2^2 + \frac{\sigma^2}{2\tau^2}||\beta||_2^2$, which matches ridge with $\lambda = \sigma^2/\tau^2$. For a Laplace prior with scale $b$, $-\log p(\beta) \propto \frac{1}{b}||\beta||_1$, giving $\frac{1}{2}||y-X\beta||_2^2 + \frac{\sigma^2}{b}||\beta||_1$, which matches lasso with $\lambda = \sigma^2/b$. This is exactly what A states, confirming it as correct. B is wrong because it swaps the priors — Laplace gives an $\ell_1$ penalty (lasso), not ridge, and the $\lambda$ expressions are inverted. C is wrong because it correctly assigns normal→ridge and Laplace→lasso but flips the $\lambda$ relationships: the correct ratios are $\sigma^2/\tau^2$ and $\sigma^2/b$, not $\tau^2/\sigma^2$ and $b/\sigma^2$. D is wrong because uniform priors contribute no penalty (flat log-prior), and point-mass priors don't produce lasso. A reliable memory aid: the shape of the prior matches the shape of the penalty — bell curve (Normal) → quadratic $\ell_2$, peaked tent (Laplace) → linear $\ell_1$. Always derive $\lambda$ by scaling the likelihood by $\sigma^2$, so it appears in the numerator of the ratio.
2. Using the one-standard-error rule, which penalty is selected, and what is the principal reason for that choice?
   Answer: $\lambda=0.30$, because it is the largest candidate within one standard error of the minimum, favoring greater regularization. Whenever you see a question involving cross-validation and model selection, think about the bias-variance tradeoff: a simpler, more regularized model may generalize better even if it doesn't achieve the absolute lowest cross-validation error, because that minimum is itself a noisy estimate. The one-standard-error rule operationalizes this idea. First, compute the threshold: minimum error $+$ one standard error $= 1.20 + 0.05 = 1.25$. Then, among all candidates with larger penalties than the minimizer (i.e., more regularized models), select the largest $\lambda$ whose cross-validation error still falls at or below that threshold. Why largest? Because a larger $\lambda$ means stronger regularization and a sparser, more interpretable model — and since its error is statistically indistinguishable from the minimum, you prefer the simpler solution. Here, $\lambda = 0.30$ has error $1.24 \leq 1.25$ ✓, while $\lambda = 1.00$ has error $1.29 > 1.25$ ✗. So C is correct: $\lambda = 0.30$ is selected precisely because it is the largest penalty still within one standard error of the minimum. Choice A is wrong because the rule doesn't compare a candidate's error to its own standard error — it compares to the threshold built from the minimum's standard error. Choice B is wrong because it ignores the rule entirely; simply keeping $\lambda = 0.08$ would mean never applying the one-standard-error principle at all. Choice D is wrong because "Monte Carlo noise" is not the stated justification — the rule is a deliberate regularization-preference criterion, not a noise-tolerance argument. Study tip: Memorize the two-step procedure — compute $\hat{\text{err}}_{\min} + 1 \cdot \widehat{SE}$, then pick the largest $\lambda$ (not smallest error) that clears that bar.
3. Which statement best compares ridge and lasso in this setting?
   Answer: Ridge uniquely assigns $β_1=β_2=s/2$, whereas lasso can have multiple nonnegative splits of the same combined coefficient. When two predictors are perfectly collinear ($x_1 = x_2$), the penalty structure of ridge versus lasso creates fundamentally different solution sets — this question tests whether you understand how each regularizer handles non-uniqueness. For ridge regression, the penalty is $\lambda(\beta_1^2 + \beta_2^2)$. Subject to a fixed sum $\beta_1 + \beta_2 = s$, this strictly convex penalty has a unique minimizer. Minimizing $\beta_1^2 + \beta_2^2$ over all splits of $s$ yields $\beta_1 = \beta_2 = s/2$ — the equal split is the only solution. The circular contours of the ridge penalty touch the constraint line at exactly one point. For lasso, the penalty is $\lambda(|\beta_1| + |\beta_2|)$. When $s > 0$, any split with $\beta_1 \geq 0$ and $\beta_2 \geq 0$ satisfying $\beta_1 + \beta_2 = s$ yields the same lasso penalty value $\lambda s$. Because the $\ell_1$ ball has a flat edge aligned with the constraint, infinitely many solutions exist — all nonnegative splits are equally optimal. This confirms A is correct. B is wrong because it reverses the roles: lasso lacks uniqueness here, not ridge. C is wrong because neither method assigns the full coefficient to one predictor — ridge always splits equally, and lasso spreads it across a continuum of nonneg solutions. D is wrong because lasso solutions with opposite-sign coefficients (e.g., $\beta_1 = s+t, \beta_2 = -t$) would increase the $\ell_1$ penalty, so they are suboptimal. Study tip: Remember the geometry — ridge has strictly convex (circular) level sets giving unique solutions, while lasso's diamond-shaped level sets have flat edges that can align with constraints, creating solution manifolds.
4. Which statement about the corresponding constrained formulation is necessarily valid?
   Answer: $β_{\lambda}$ solves the least-squares problem subject to $||β||_1\le t$, although the constrained coefficient solution need not be unique. Whenever you see a question linking penalized and constrained forms of the lasso, you should think about the Lagrangian duality connecting them: the penalized problem $\min_\beta \{\frac{1}{2}\|y-X\beta\|_2^2 + \lambda\|\beta\|_1\}$ and the constrained problem $\min_\beta \frac{1}{2}\|y-X\beta\|_2^2 \text{ subject to } \|\beta\|_1 \le t$ are equivalent in a precise sense via KKT conditions. For any solution $\beta_\lambda$ of the penalized problem, setting $t = \|\beta_\lambda\|_1$ means $\beta_\lambda$ is feasible for the constrained problem. The KKT conditions for both problems are satisfied by the same point with the penalty parameter $\lambda$ acting as the Lagrange multiplier. Therefore $\beta_\lambda$ is indeed a solution to the constrained problem at that value of $t$. However, because the objective $\frac{1}{2}\|y - X\beta\|_2^2$ is convex but not strictly convex (e.g., when $X$ is rank-deficient or columns are collinear), the constrained problem may have multiple optimal solutions — $\beta_\lambda$ is a solution, not necessarily the unique one. This makes B correct. A is wrong because the constraint bound $t = \|\beta_\lambda\|_1$ need not equal $\lambda$; these are dimensionally different objects and the relationship between them depends on the data. C is wrong because the $\ell_1$ penalty does not make the full objective strictly convex in $\beta$ — strict convexity requires the loss term alone to be strictly convex, which fails when $X$ is not full column rank. D is wrong because the equivalence between penalized and constrained forms holds generally, without requiring full column rank or nonzero entries. As a study tip: always distinguish between existence (guaranteed by convexity) and uniqueness (requires strict convexity) of lasso solutions — exam questions frequently conflate the two.
5. What are the expectation and variance of the ridge estimate along this eigenvector?
   Answer: Expectation $2.4$ and variance $0.32$ When analyzing ridge regression along a single eigenvector, you should work in the rotated coordinate system where $X^TX$ is diagonal. Along one eigenvector with eigenvalue $d$, the ridge estimator shrinks the OLS estimate by the factor $\frac{d}{d+\lambda}$, which creates bias but reduces variance. Expectation: The ridge estimator is biased. Its expectation along this eigenvector is the true coefficient multiplied by the shrinkage factor: $E[\hat{\beta}_\lambda] = \frac{d}{d+\lambda}\beta = \frac{4}{4+1}(3) = \frac{4}{5}(3) = 2.4$. This confirms the expectation is $2.4$. Variance: The variance of the OLS estimator along this eigenvector is $\frac{\sigma^2}{d} = \frac{2}{4} = 0.5$. Ridge applies the shrinkage factor squared to this variance: $\text{Var}(\hat{\beta}_\lambda) = \left(\frac{d}{d+\lambda}\right)^2 \frac{\sigma^2}{d} = \left(\frac{4}{5}\right)^2(0.5) = 0.64 \times 0.5 = 0.32$. So the correct answer is C: expectation $2.4$ and variance $0.32$. Choice A gets the expectation right but uses the unsquared shrinkage factor on the variance: $0.8 \times 0.5 = 0.40$, a common computational slip. Choice B incorrectly reports the expectation as $3.0$, which would only hold for OLS (no shrinkage), ignoring ridge's inherent bias entirely. Choice D applies the shrinkage factor to the variance without first computing the OLS variance correctly, arriving at implausibly small values. Your study tip: always remember that ridge bias uses the shrinkage factor once, but the variance uses it squared — these two formulas are easy to conflate under exam pressure.
6. Which pair gives the ridge estimate followed by the lasso estimate?
   Answer: $(1.5,0.25,-1)^T$ and $(2,0,-1)^T$ When $X^TX = I$, the OLS estimate simplifies beautifully to $\hat{\beta}_{OLS} = X^Ty = z = (3, 0.5, -2)^T$. Both ridge and lasso then reduce to coordinate-wise shrinkage problems, which makes this question entirely about recognizing two distinct shrinkage operators. Ridge with the given objective adds $\frac{\lambda}{2}\|\beta\|_2^2$, yielding the closed-form solution $\hat{\beta}_{ridge} = \frac{1}{1+\lambda}z$. With $\lambda = 1$, each coordinate is multiplied by $\frac{1}{2}$: $(3, 0.5, -2)^T \to (1.5, 0.25, -1)^T$. Ridge shrinks every coefficient proportionally toward zero — none are set exactly to zero. Lasso applies soft-thresholding: $\hat{\beta}_{lasso,j} = \text{sign}(z_j)(|z_j| - \lambda)_+$. With $\lambda = 1$: the first coordinate gives $\text{sign}(3)(3-1) = 2$, the second gives $\text{sign}(0.5)(0.5-1)_+ = 0$ (thresholded to zero), and the third gives $\text{sign}(-2)(2-1) = -1$. So the lasso estimate is $(2, 0, -1)^T$. Lasso's key feature — exact zeros via thresholding — eliminates the small coefficient entirely. The correct answer is A: ridge $(1.5, 0.25, -1)^T$ followed by lasso $(2, 0, -1)^T$. Choice B reverses the two estimates. Choice C mixes them incorrectly, preserving the lasso's zero in the ridge solution. Choice D incorrectly zeros out the ridge estimate while leaving the lasso coefficient at its raw OLS value. Your study tip: memorize the two operators — ridge scales by $\frac{1}{1+\lambda}$, lasso soft-thresholds by $\lambda$. Under orthonormal designs, these apply independently per coordinate, making calculation straightforward.
7. What is the effective degrees of freedom of the ridge fit, defined as the trace of its smoothing matrix?
   Answer: $1.4$, obtained by summing the three ridge shrinkage factors Whenever you encounter a question about ridge regression degrees of freedom, anchor your thinking to the spectral decomposition of the hat matrix. The ridge smoothing matrix is $H_\lambda = X(X^TX + \lambda I)^{-1}X^T$, and its trace — the effective degrees of freedom — equals $\sum_{j} \frac{d_j^2}{d_j^2 + \lambda}$, where $d_j$ are the singular values of $X$. Here, the squared singular values are $d_1^2 = 9$, $d_2^2 = 1$, and $d_3^2 = 0$, with $\lambda = 1$. Plugging in: $\text{df}(\lambda) = \frac{9}{9+1} + \frac{1}{1+1} + \frac{0}{0+1} = 0.9 + 0.5 + 0.0 = 1.4$ This confirms D is correct. Choice A (3.0) would be correct only for OLS with a full-rank $3 \times 3$ system — ridge shrinkage strictly reduces effective df below the number of coefficients. Choice B (2.0) reflects the rank of $X^TX$ (two nonzero eigenvalues), which is the OLS effective df for this rank-2 matrix — it ignores the penalty entirely. Choice C (1.5) comes from naively averaging the two nonzero contributions $(0.9 + 0.5)/2 \times 2 = 1.5$... actually it simply averages them as $\frac{0.9+0.5}{2} \approx 0.7$, then doubles — a tempting but unjustified manipulation. The correct operation is a sum, not a rescaled average. Your study tip: memorize the formula $\text{df}(\lambda) = \sum_j \frac{d_j^2}{d_j^2+\lambda}$ cold. It appears frequently in theory questions, and the most common trap is substituting the rank or the coefficient count instead of computing the shrinkage-weighted sum.
8. What is the updated value of $β_j$?
   Answer: $-0.06$, because soft thresholding reduces the magnitude by $0.20$ Whenever you see a lasso coordinate-descent question, your mental anchor should be the soft-thresholding operator. Because $x_j^Tx_j/n = 1$ (standardization), the closed-form update for $\beta_j$ is simply: $\beta_j = S\!\left(\frac{x_j^T r_j}{n},\, \lambda\right) = \text{sign}(z)\cdot\max(|z| - \lambda,\, 0)$ where $z = x_j^T r_j / n$. Plugging in $z = -0.26$ and $\lambda = 0.20$: $\beta_j = \text{sign}(-0.26)\cdot\max(0.26 - 0.20,\, 0) = (-1)(0.06) = -0.06$ The penalty shrinks the magnitude by exactly $\lambda$, giving D as the correct answer. Choice A claims the penalty is "divided across two signs," which has no basis in lasso theory — soft thresholding subtracts the full $\lambda$ from the magnitude, not half of it. Choice B is a tempting trap: standardization does simplify the update formula, but it does not eliminate the penalty term. The unpenalized OLS update would be $-0.26$; the lasso update is always shrunk toward zero. Choice C would be correct only if $|z| \leq \lambda$, which would force $\beta_j = 0$. Here $0.26 > 0.20$, so the coefficient survives shrinkage — it's reduced but not zeroed out. Study tip: Memorize the soft-thresholding rule as "subtract $\lambda$ from the absolute value, clip at zero, restore the sign." On exam questions, always check whether $|z|$ exceeds $\lambda$ first — that one comparison tells you immediately whether the answer is zero or a shrunken nonzero value.
9. Which statement about uniqueness is generally correct?
   Answer: Ridge coefficients are unique; lasso coefficients may be nonunique, although all lasso solutions have the same fitted values on the training design. When you see a question combining penalized regression with rank-deficient design matrices, your instinct should be to separately analyze the loss geometry and the penalty geometry for each method. Ridge regression adds an $\ell_2$ penalty: $\lambda \|\beta\|_2^2$. This term is strictly convex and smooth, and when added to any convex loss, the entire objective becomes strictly convex — regardless of whether $X^TX$ is rank-deficient. A strictly convex function has exactly one minimizer, so ridge coefficients are always unique, even when $p > n$ or the design matrix is rank-deficient. This confirms answer B's first clause. Lasso adds an $\ell_1$ penalty: $\lambda \|\beta\|_1$. This is convex but not strictly convex — it is piecewise linear. Adding a non-strictly-convex penalty to a non-strictly-convex loss (squared error with rank-deficient $X$) leaves open the possibility of multiple minimizers. Critically, even when multiple lasso solutions exist, they all produce identical fitted values $X\hat{\beta}$ on the training data, because $X\beta$ must lie in a convex set where the loss is constant. This confirms B's second clause. A is wrong because strict convexity of the full objective requires the penalty itself to be strictly convex — the $\ell_1$ penalty is not, so this reasoning fails for lasso. C is wrong because it incorrectly assumes rank deficiency breaks uniqueness for ridge; the $\ell_2$ penalty rescues strict convexity regardless. D reverses the correct pairing entirely — it's ridge, not lasso, that guarantees unique coefficients. Your study tip: memorize that $\ell_2$ penalties confer strict convexity → unique coefficients; $\ell_1$ penalties do not → potential non-uniqueness, but fitted values remain unique.
10. How does this rescaling affect the role of that predictor under ordinary ridge or lasso penalization?
    Answer: It makes the predictor effectively less penalized because an equivalent fitted effect requires a coefficient one-tenth as large. Whenever you see a question about penalized regression and rescaling, your first instinct should be to ask: what does the penalty actually operate on? Both ridge and lasso penalize the coefficients, not the predictors directly. Ridge penalizes $\sum_j \beta_j^2$ and lasso penalizes $\sum_j |\beta_j|$, each multiplied by $\lambda$. Here's the key insight. Suppose the true relationship is $\beta_j x_j$. If you replace $x_j$ with $x_j^* = 10x_j$, then to produce the same fitted contribution to the model, you only need $\beta_j^* = \beta_j / 10$. A coefficient ten times smaller is subject to far less penalization — under ridge, $(\beta_j/10)^2$ is 100 times smaller than $\beta_j^2$; under lasso, $|\beta_j/10|$ is ten times smaller. So the penalty barely restrains this predictor compared to others on their original scales. Answer A is correct: rescaling $x_j$ by a factor of 10 makes it effectively less penalized, because representing the same effect requires a proportionally smaller coefficient. Answer B gets the direction backwards — larger observed values actually allow a smaller coefficient, reducing, not increasing, the penalty burden. Answer C is the most tempting trap: it's true that the optimal fitted model is theoretically scale-invariant, but only if $\lambda$ is adjusted accordingly. Using the same $\lambda$ breaks that invariance entirely. Answer D incorrectly claims ridge is unaffected; both penalties operate on coefficient magnitude and are equally disrupted by the fixed $\lambda$. Study tip: Always remember that standardizing predictors before penalized regression ensures each coefficient is penalized on a fair, comparable scale — this is precisely why standardization is considered best practice.

Start this quiz