Inference for Regression Coefficients

SECTION 1

Historical Context & Motivation

The problem of fitting a line to observed data is surprisingly old, but the question of whether that fitted line reveals a genuine relationship or merely an artifact of sampling variability required decades of theoretical development. Early astronomers and geodesists knew how to estimate linear relationships by eye or by minimizing residuals, yet they lacked a rigorous framework for distinguishing signal from noise. The intellectual journey from curve-fitting to formal statistical inference for regression coefficients spans two centuries and draws on probability theory, sampling distributions, and the theory of estimation.

1805

Legendre's Method of Least Squares

Adrien-Marie Legendre publishes the first formal description of the method of least squares, providing a systematic algorithm for fitting linear models to data. Gauss later claims independent discovery, supplying a probabilistic justification under normality.

1908

Student's t-Distribution

William Sealy Gosset, writing under the pseudonym 'Student,' derives the t-distribution for small-sample inference when the population variance is unknown — the key distribution for testing individual regression coefficients.

1922

Fisher's Maximum Likelihood Framework

R. A. Fisher formalizes maximum likelihood estimation and introduces the concept of sufficient statistics, providing a theoretical basis for the efficiency and optimality of OLS estimators under normality.

1935

The Gauss–Markov Theorem

The Gauss–Markov theorem is rigorously established, proving that the OLS estimator is the best linear unbiased estimator (BLUE) under the classical assumptions — even without normality — cementing the theoretical foundation for regression inference.

1970s

Robust and Heteroscedasticity-Consistent Inference

Halbert White and others develop heteroscedasticity-consistent standard errors, extending inference for regression coefficients beyond the strict classical assumptions and enabling valid tests under model misspecification.

The central question motivating this lesson is deceptively simple: given an estimated regression coefficient β̂, how do we determine whether the corresponding predictor has a statistically significant relationship with the response, and how precisely can we estimate the magnitude of that relationship? Answering this requires understanding the sampling distribution of β̂, constructing standard errors, formulating t-tests, and building confidence intervals — all of which rest on the classical linear model assumptions.

SECTION 2

Core Principles & Definitions

Inference for regression coefficients rests on several interconnected ideas that bridge the gap between point estimation and formal hypothesis testing. At its core, the enterprise recognizes that every estimated coefficient is a random variable — its numerical value would change if we collected a different sample from the same population. To make probability statements about the true parameter, we need to characterize the distribution of the estimator under repeated sampling, quantify its variability through standard errors, and then leverage known reference distributions to build tests and intervals.

Sampling Distribution of β̂

Under the classical assumptions (linearity, exogeneity, homoscedasticity, normality), the OLS estimator β̂ follows a multivariate normal distribution centered at the true parameter vector β, with covariance matrix σ²(X⊤X)⁻¹.

Standard Error

The standard error of β̂ⱼ is the estimated standard deviation of its sampling distribution. It equals σ̂ times the square root of the j-th diagonal element of (X⊤X)⁻¹, capturing how precisely we can estimate βⱼ.

t-Statistic for Individual Coefficients

The ratio (β̂ⱼ − βⱼ₀) / SE(β̂ⱼ) follows a t-distribution with n − p degrees of freedom, where βⱼ₀ is the null hypothesis value. This pivotal quantity is the basis for all single-coefficient tests.

Confidence Intervals

A (1 − α) × 100% confidence interval for βⱼ is β̂ⱼ ± t_{α/2, n−p} × SE(β̂ⱼ). Over repeated samples, approximately (1 − α) × 100% of such intervals contain the true parameter — a frequentist coverage guarantee.

F-Test for Groups of Coefficients

To test whether a subset of coefficients are simultaneously zero, we use the general linear F-test, which compares the residual sums of squares from the full and reduced models. This generalizes the individual t-test to multiple constraints.

✦ KEY TAKEAWAY

Think of estimating a regression coefficient like measuring the gravitational constant with a particular experimental apparatus. Each time you run the experiment, you get a slightly different reading due to measurement noise. The standard error quantifies the typical spread of those readings, and a confidence interval tells you the plausible range for the true constant given your particular measurement and its precision.

SECTION 3

Visual Explanation

The following diagram illustrates the conceptual pipeline of inference for a single regression coefficient β̂ⱼ. Starting from the observed data and the OLS estimate, we construct a sampling distribution by characterizing the estimator's variability, then form a t-statistic and map it onto the t-distribution to obtain p-values and confidence intervals. The diagram highlights how the classical assumptions feed into each stage of the inferential chain.

The inference pipeline starts with observed data, produces the OLS estimate β̂ⱼ, computes its standard error using σ̂ and (X⊤X)⁻¹, and forms a t-statistic that is compared to the t-distribution with n − p degrees of freedom. The shaded tails represent the rejection region for a two-sided test at significance level α. The dashed green box indicates how the classical assumptions underpin each stage.

Notice how the pipeline has two distinct logical threads. The estimation thread (top row, left to right) computes numerical quantities from the data, while the distributional thread (dashed green box and the t-curve below) provides the probability model that makes the numerical quantities interpretable as evidence. Without the classical assumptions justifying the distributional thread, the t-statistic is merely a ratio with no known reference distribution, and formal inference breaks down.

SECTION 4

Mathematical Framework

We work within the classical normal linear model: y = Xβ + ε, where y is an n × 1 response vector, X is an n × p design matrix of full column rank, β is the p × 1 parameter vector, and ε ~ N(0, σ²Iₙ). The OLS estimator is β̂ = (X⊤X)⁻¹X⊤y. Under these assumptions, we can derive exact distributional results for β̂ and construct pivotal quantities for inference.

DISTRIBUTION OF THE OLS ESTIMATOR

β̂ ~ N(β, σ²(X⊤X)⁻¹)

This follows because β̂ = β + (X⊤X)⁻¹X⊤ε is a linear transformation of the normal vector ε. The covariance matrix σ²(X⊤X)⁻¹ shows that the precision of each estimator depends on both the error variance σ² and the geometry of the design matrix X.

UNBIASED ESTIMATOR OF σ²

σ̂² = RSS / (n − p) = e⊤e / (n − p)

Here e = y − Xβ̂ is the vector of residuals and RSS is the residual sum of squares. The divisor n − p accounts for the p degrees of freedom consumed by estimating β. It can be shown that (n − p)σ̂² / σ² ~ χ²(n − p), and that σ̂² is independent of β̂ — a crucial fact for constructing the t-statistic.

t-STATISTIC FOR A SINGLE COEFFICIENT

tⱼ = (β̂ⱼ − βⱼ₀) / SE(β̂ⱼ) = (β̂ⱼ − βⱼ₀) / (σ̂ √cⱼⱼ) ~ t(n − p)

Here cⱼⱼ is the j-th diagonal element of (X⊤X)⁻¹, and βⱼ₀ is the hypothesized value (typically 0). The t-distribution arises because we divide a standard normal random variable (β̂ⱼ − βⱼ) / (σ √cⱼⱼ) by √(σ̂²/σ²), where the latter is the square root of a χ²/(n − p) random variable that is independent of the numerator.

CONFIDENCE INTERVAL FOR βⱼ

β̂ⱼ ± t_{α/2, n−p} × SE(β̂ⱼ)

This interval has exact (1 − α) coverage probability under the normal linear model. The critical value t_{α/2, n−p} is the upper α/2 quantile of the t-distribution with n − p degrees of freedom. As n − p → ∞, t_{α/2, n−p} → z_{α/2}, and the interval approaches the z-interval.

📐 Derivation Note: Independence of β̂ and σ̂²

The independence of β̂ and σ̂² is a consequence of Cochran's theorem. Since β̂ = (X⊤X)⁻¹X⊤y lies in the column space of X and the residual e = (I − H)y lies in its orthogonal complement (where H = X(X⊤X)⁻¹X⊤ is the hat matrix), they are functions of independent projections of the normal vector y and are therefore independent. This independence is essential for the ratio tⱼ to follow an exact t-distribution.

SECTION 5

Anatomy of the Standard Error

The standard error of β̂ⱼ deserves careful dissection because it controls the width of confidence intervals, the magnitude of t-statistics, and ultimately the power of hypothesis tests. Understanding what makes a standard error small or large is essential for both interpreting regression output and designing studies that can detect meaningful effects.

The four primary factors influencing the standard error of β̂ⱼ: error variance σ², sample size n, the variability of predictor xⱼ (SSxⱼ), and multicollinearity as measured by the variance inflation factor VIFⱼ. The combined formula at the bottom right expresses SE(β̂ⱼ) as σ̂ divided by √(SSxⱼ(1 − Rⱼ²)), where Rⱼ² is the R² from regressing xⱼ on all other predictors.

In the simple linear regression case with a single predictor x, the diagonal element cⱼⱼ simplifies to 1/SSₓ, where SSₓ = Σ(xᵢ − x̄)². The standard error thus becomes σ̂/√SSₓ, making the dependence on predictor spread transparent. In the multiple regression setting, multicollinearity inflates cⱼⱼ beyond 1/SSₓⱼ by a multiplicative factor of VIFⱼ = 1/(1 − Rⱼ²), where Rⱼ² is the R-squared from an auxiliary regression of xⱼ on the remaining predictors. A VIF of 10, for example, inflates the standard error by √10 ≈ 3.16 compared to the no-collinearity benchmark — dramatically reducing the power of the t-test for that coefficient.

Summary of factors affecting the standard error of β̂ⱼ

Factor	Effect on SE(β̂ⱼ)	Practical Implication
σ² (error variance)	Directly proportional (SE ∝ σ̂)	Reducing measurement error or adding explanatory variables lowers σ̂² and tightens inference.
n (sample size)	Inversely proportional (SE ∝ 1/√n approximately)	Doubling sample size reduces SE by ≈ 30%, not 50% — diminishing marginal returns.
Var(xⱼ) (predictor spread)	Inversely proportional (SE ∝ 1/√SSₓⱼ)	In experiments, spacing design points widely (e.g., factorial designs) maximizes precision.
VIFⱼ (multicollinearity)	Directly proportional (SE ∝ √VIFⱼ)	High collinearity inflates SE, making it difficult to distinguish individual predictor effects.

SECTION 6

Worked Example

Consider a study examining the relationship between years of education (x₁), years of work experience (x₂), and annual salary (y, in thousands of dollars) for n = 30 individuals. The fitted multiple regression model yields β̂₀ = 15.2, β̂₁ = 4.35, β̂₂ = 1.12, with residual standard error σ̂ = 5.80 (based on n − p = 30 − 3 = 27 degrees of freedom). We also have SE(β̂₁) = 0.82 and SE(β̂₂) = 0.48. We wish to test whether education has a statistically significant positive effect on salary and construct a 95% confidence interval for β₁.

Testing and Estimating the Education Coefficient β₁

Step 1 — State the HypothesesWe test H₀: β₁ = 0 (education has no linear effect on salary, controlling for experience) against H₁: β₁ ≠ 0 (education does have a linear effect). Although our research question suggests a one-sided test (positive effect), we proceed with a two-sided test to be conservative and then note that the one-sided p-value is half the two-sided one.

Step 2 — Compute the t-StatisticThe t-statistic for β₁ is: t₁ = (β̂₁ − 0) / SE(β̂₁) = 4.35 / 0.82 = 5.305. This tells us the estimated coefficient is approximately 5.3 standard errors away from the null value of zero.

t₁ = 5.305

Step 3 — Determine the Reference Distribution and p-valueUnder H₀, t₁ follows a t-distribution with n − p = 27 degrees of freedom. The critical value for a two-sided test at α = 0.05 is t₀.₀₂₅,₂₇ ≈ 2.052. Since |t₁| = 5.305 >> 2.052, we are deep in the rejection region. Using a t-table or software, the two-sided p-value is approximately P(|t₂₇| > 5.305) ≈ 0.000014.

p ≈ 0.000014 (two-sided)

Step 4 — Make a DecisionSince the p-value (0.000014) is far below α = 0.05, we reject H₀. There is very strong evidence that education has a nonzero linear effect on salary, controlling for years of experience. For the one-sided alternative H₁: β₁ > 0, the one-sided p-value ≈ 0.000007, and since β̂₁ > 0, the result supports a positive association.

Step 5 — Construct a 95% Confidence Interval for β₁The 95% CI is: β̂₁ ± t₀.₀₂₅,₂₇ × SE(β̂₁) = 4.35 ± 2.052 × 0.82 = 4.35 ± 1.683. This yields the interval (2.667, 6.033). We are 95% confident that each additional year of education is associated with an increase in annual salary between approximately $2,667 and $6,033, holding work experience constant.

95% CI for β₁: (2.667, 6.033) thousand dollars per year of education

✓ Consistency Check

Notice that the 95% CI does not contain zero, which is consistent with rejecting H₀: β₁ = 0 at the α = 0.05 level. This duality — a CI excluding β₁₀ is equivalent to rejecting H₀: β₁ = β₁₀ at the corresponding significance level — is a fundamental property of the relationship between tests and intervals built from the same pivotal t-statistic.

SECTION 7

Assumptions, Robustness, and Diagnostics

The exact t- and F-tests derived in Section 4 rely on the classical assumptions. In practice, one or more assumptions may be violated. Understanding which violations are mild (inference remains approximately valid) and which are severe (inference can be misleading) is critical for responsible applied work. The table below summarizes the robustness profile of standard regression inference.

Robustness of regression inference to assumption violations

Assumption	Consequence if Violated	Diagnostic / Fix
Linearity	Biased β̂; SEs and tests are invalid because the model is misspecified. E[β̂] ≠ β due to omitted nonlinear terms.	Residual vs. fitted plots, added-variable plots, RESET test. Fix with transformations, polynomials, or splines.
Independence	OLS remains unbiased, but SEs are incorrect (usually understated), leading to inflated Type I error rates.	Durbin–Watson test (time series); clustered SEs; mixed models; GEE for longitudinal data.
Homoscedasticity	OLS is still unbiased but no longer BLUE. Classical SEs are wrong; CIs may be too narrow or too wide.	Breusch–Pagan test, White test. Fix with White (HC) robust SEs, WLS, or variance-stabilizing transforms.
Normality of ε	t-tests and F-tests are only approximate. For moderate n, the CLT makes this mild; for small n, nonnormality can distort p-values.	Q–Q plot of residuals, Shapiro–Wilk test. Fix with transformations, robust regression, or bootstrap inference.
No perfect multicollinearity	(X⊤X) is singular; OLS cannot be computed. Near-multicollinearity inflates SEs dramatically.	VIF > 10 is a common rule of thumb. Fix by dropping variables, PCA, or ridge regression.

✦ KEY TAKEAWAY

Think of the classical assumptions like the conditions listed on a warranty. If you follow all the conditions exactly, the manufacturer (the theory) guarantees the product (your t-test) works precisely as specified. In practice, small deviations — a slightly dented box, mild heteroscedasticity — rarely void the warranty thanks to the Central Limit Theorem acting as a robustness buffer. But major violations — like using the product underwater (severe misspecification) — require fundamentally different tools: robust standard errors, bootstrap methods, or generalized linear models.

SECTION 8

Connection to Generalized Linear Models

The inferential framework for regression coefficients extends naturally to generalized linear models (GLMs), where the response may be non-normal (e.g., binary, count, or gamma-distributed). The key difference is that exact t- and F-distributions are no longer available; instead, inference is based on asymptotic theory — specifically, the Wald, likelihood ratio, and score tests, all of which converge to the same chi-squared or normal reference distribution as n → ∞. Understanding the classical normal linear model case is essential because it is the one setting where these asymptotics are exact in finite samples.

Classical OLS inference vs. GLM inference

Feature	Classical Linear Model	Generalized Linear Model
Estimation method	OLS (closed-form solution)	Maximum likelihood via IRLS (iterative)
Sampling distribution of β̂	Exact: N(β, σ²(X⊤X)⁻¹)	Asymptotic: N(β, I(β)⁻¹) where I(β) is the Fisher information
Single-coefficient test	t-test (exact under assumptions)	Wald z-test: z = β̂ⱼ / SE(β̂ⱼ) ~ N(0,1) asymptotically
Multi-coefficient test	F-test (exact)	Likelihood ratio test: −2 log Λ ~ χ²(q) asymptotically
Confidence intervals	β̂ⱼ ± t_{α/2,n−p} × SE (exact coverage)	β̂ⱼ ± z_{α/2} × SE (approximate); profile likelihood CIs preferred
Nuisance variance parameter	σ² estimated separately; impacts df	May be known (Poisson, binomial) or estimated (negative binomial, quasi-likelihood)

In the GLM framework, the Wald test is the direct analogue of the t-test: it divides the estimated coefficient by its estimated asymptotic standard error. However, for small samples, the Wald test can be unreliable (especially with logistic regression near boundary cases), and the likelihood ratio test is generally preferred because it has better finite-sample properties. Mastering inference in the normal linear model — where everything is exact — gives you the conceptual scaffolding to recognize what is gained and lost when moving to asymptotic inference in more complex settings.

SECTION 9

Practice Problems

PROBLEM 1 — CONCEPTUAL

Explain why the t-statistic for testing H₀: βⱼ = 0 follows a t-distribution rather than a standard normal distribution. Under what condition does the t-distribution converge to the z-distribution, and what is the practical implication of this convergence for large datasets?

PROBLEM 2 — BASIC CALCULATION

A simple linear regression of y on x with n = 25 observations yields β̂₁ = 3.20 and SE(β̂₁) = 1.15. Compute the t-statistic for testing H₀: β₁ = 0, identify the degrees of freedom, and determine whether you reject H₀ at the α = 0.05 level (two-sided). Use t₀.₀₂₅,₂₃ ≈ 2.069.

PROBLEM 3 — INTERMEDIATE

In a multiple regression with n = 50 and p = 5 (including the intercept), the residual sum of squares is RSS = 360. The j-th diagonal element of (X⊤X)⁻¹ is cⱼⱼ = 0.025. Compute σ̂², SE(β̂ⱼ), and the width of a 95% confidence interval for βⱼ. Use t₀.₀₂₅,₄₅ ≈ 2.014.

PROBLEM 4 — APPLIED

A health researcher fits a multiple linear regression predicting systolic blood pressure (mmHg) from age (years), BMI (kg/m²), and sodium intake (mg/day) for n = 120 patients. The coefficient for sodium intake is β̂₃ = 0.0038 with SE(β̂₃) = 0.0015, and the VIF for sodium intake is 3.8. (a) Test H₀: β₃ = 0 at α = 0.01 (use z₀.₀₀₅ ≈ 2.576 as an approximation given large n). (b) If the VIF were reduced to 1.0, what would the new standard error be approximately? (c) Discuss the practical significance of the estimated effect.

PROBLEM 5 — CRITICAL THINKING

Prove that the (1 − α) confidence interval β̂ⱼ ± t_{α/2, n−p} × SE(β̂ⱼ) is equivalent to the set of all values βⱼ₀ for which the two-sided t-test of H₀: βⱼ = βⱼ₀ would not be rejected at level α. Then, discuss how this duality breaks down in the GLM setting when Wald-based and likelihood ratio–based confidence intervals do not coincide.

SUMMARY

Summary

Inference for regression coefficients begins with the OLS estimator β̂ = (X⊤X)⁻¹X⊤y, whose sampling distribution under the classical normal linear model is exactly N(β, σ²(X⊤X)⁻¹). The standard error of β̂ⱼ, given by σ̂√cⱼⱼ, captures uncertainty arising from error variance, sample size, predictor spread, and multicollinearity (VIF). The t-statistic tⱼ = β̂ⱼ/SE(β̂ⱼ) follows a t(n − p) distribution under H₀, enabling exact hypothesis tests and confidence intervals of the form β̂ⱼ ± t_{α/2, n−p} × SE(β̂ⱼ), with guaranteed (1 − α) coverage.

The validity of these procedures depends on the classical assumptions — linearity, independence, homoscedasticity, and normality — though large-sample robustness via the CLT provides protection against mild violations. When assumptions are seriously violated, heteroscedasticity-consistent standard errors, bootstrap methods, or generalized linear models provide alternative inferential frameworks. In the GLM setting, the exact t-test gives way to asymptotic Wald, likelihood ratio, and score tests — a natural generalization that preserves the core logic of comparing an estimated effect to its estimated variability.