Simple Linear Regression — Fit and interpret simple linear regression in context - Biostatistics

SECTION 1

Historical Context & Motivation

The desire to describe how one quantity changes in response to another is among the oldest problems in quantitative science. Long before the formal machinery of simple linear regression was codified, astronomers and natural philosophers plotted observations against predictions and searched for the "best" straight line through a scatter of points. The statistical framework that emerged from these efforts now underpins virtually every branch of the life sciences, from dose–response pharmacology to genome-wide association studies. Understanding its origins clarifies why the method takes the particular form it does and reveals the assumptions baked into every regression table produced by modern software.

1805 • Legendre Publishes the Method of Least Squares

Adrien-Marie Legendre formalized the least-squares criterion in his work on comet trajectories, proposing that the best-fit line minimizes the sum of squared deviations between observed and predicted values.

1809 • Gauss Derives the Normal Error Model

Carl Friedrich Gauss independently derived the least-squares estimator and showed that it is optimal when errors follow a normal distribution, establishing the probabilistic foundation for regression inference.

1886 • Galton Coins 'Regression'

Francis Galton studied the relationship between parents' and children's heights, observing that extreme parental heights tended to produce offspring closer to the mean—a phenomenon he called regression toward mediocrity. The term 'regression' persists today.

1922 • Fisher Formalizes Inference for Regression

Ronald A. Fisher unified estimation and hypothesis testing within the regression framework, introducing the analysis of variance (ANOVA) decomposition and t-tests for regression coefficients that remain standard practice.

1970s–present • Computational Revolution

The advent of electronic computing and software packages such as SAS, SPSS, Stata, and R made regression analysis accessible to every biomedical researcher, transforming it into the default first-pass analytic tool in clinical and epidemiological research.

The central question simple linear regression answers is deceptively straightforward: given a set of paired observations, what is the best straight-line summary of how the outcome variable changes, on average, for each one-unit increase in the predictor? Answering this question rigorously requires fitting the line, quantifying uncertainty around the estimated slope and intercept, and—critically—interpreting those estimates in the specific context of the biological or clinical problem at hand.

SECTION 2

Core Principles & Definitions

Before fitting a regression model, it is essential to internalize several foundational ideas that govern both the mechanics of estimation and the validity of subsequent inference. Simple linear regression relates a single continuous predictor (also called the independent variable or explanatory variable, denoted X) to a single continuous response (also called the dependent variable or outcome, denoted Y). The model posits that the expected value of Y is a linear function of X, with individual observations scattering around that line due to random error.

Linearity

The conditional mean of Y given X is a straight line: E(Y | X) = β₀ + β₁X. If the true relationship is curvilinear, the linear model will be systematically biased.

Independence

Observations are sampled independently of one another. Violations occur in longitudinal, clustered, or spatial data, where correlated errors can inflate Type I error rates.

Normality of Residuals

At each value of X, the residuals ε follow a normal distribution. This assumption is most critical for small samples; by the Central Limit Theorem, moderate to large samples are more robust.

Homoscedasticity

The variance of the residuals is constant across all levels of X (equal spread). When the spread fans out or contracts—called heteroscedasticity—standard errors and tests become unreliable.

Fixed-X / Measurement-Error-Free Predictor

Classical regression assumes X is measured without error (or at least that measurement error in X is negligible). Substantial predictor measurement error biases β̂₁ toward zero (attenuation bias).

✦

KEY TAKEAWAY

Think of simple linear regression like drawing the straightest possible path through a field of scattered data points. The least-squares line is the path that minimizes the total squared vertical distance from every point to the line—analogous to a surveyor laying a road that minimizes total detour for all nearby villages. The assumptions (linearity, independence, normality, equal variance) are the terrain conditions that must hold for that road to be the most efficient route.

SECTION 3

Visual Explanation — The Scatter Plot and Least-Squares Line

A scatter plot is the natural starting point for any regression analysis. Each point represents one observation with coordinates (xᵢ, yᵢ). The fitted regression line, denoted ŷ = b₀ + b₁x, is superimposed on the scatter, and the vertical distances from each point to the line—called residuals (eᵢ = yᵢ − ŷᵢ)—illustrate the quantity that least squares minimizes. The diagram below shows a hypothetical dataset relating daily caloric intake (X) to systolic blood pressure (Y) among 12 adults.

Each violet dot is one subject's observed (caloric intake, systolic BP) pair. The cyan line is the ordinary least-squares fit. The pink dashed segments show residuals—vertical distances the least-squares algorithm minimizes in aggregate by minimizing Σeᵢ².

Notice that the line does not pass through every point; instead it captures the average trend. Some residuals are positive (points above the line) and others negative (points below). The ordinary least-squares (OLS) procedure finds the unique line for which the sum of squared residuals, Σeᵢ², is as small as possible. Because the line always passes through the point (x̄, ȳ), the residuals necessarily sum to zero—a built-in property of the OLS fit.

SECTION 4

Mathematical Framework

The population regression model expresses the relationship between X and Y in terms of fixed (unknown) parameters and a stochastic error component. From a sample of n paired observations, we estimate these parameters using the least-squares criterion.

POPULATION REGRESSION MODEL

Yᵢ = β₀ + β₁Xᵢ + εᵢ , εᵢ ~ N(0, σ²)

β₀ = population intercept (mean of Y when X = 0); β₁ = population slope (change in E(Y) per one-unit increase in X); εᵢ = random error for observation i, assumed normally distributed with mean 0 and constant variance σ².

OLS SLOPE ESTIMATOR

b₁ = Σ(xᵢ − x̄)(yᵢ − ȳ) / Σ(xᵢ − x̄)² = S_xy / S_xx

S_xy = sample covariance numerator (sum of cross-products of deviations); S_xx = sum of squared deviations of X. The slope quantifies the average change in Y per unit change in X.

OLS INTERCEPT ESTIMATOR

b₀ = ȳ − b₁ · x̄

Because the fitted line must pass through (x̄, ȳ), the intercept is fully determined once the slope is known. In many biostatistical contexts, the intercept has no direct biological meaning (e.g., when X = 0 is outside the observed range).

COEFFICIENT OF DETERMINATION

R² = 1 − (SS_res / SS_tot) = SS_reg / SS_tot

SS_tot = Σ(yᵢ − ȳ)² (total variability); SS_res = Σ(yᵢ − ŷᵢ)² (residual variability); SS_reg = Σ(ŷᵢ − ȳ)² (explained variability). R² ranges from 0 to 1 and represents the proportion of total variance in Y explained by X.

ℹ

Standard Error of the Slope

The precision of b₁ depends on both the residual variance and the spread of the predictor: SE(b₁) = √[MS_res / S_xx], where MS_res = SS_res / (n − 2). A wider range of X values and a larger sample both reduce SE(b₁), yielding more precise slope estimates.

SECTION 5

The ANOVA Decomposition & Residual Diagnostics

One of the most illuminating ways to understand a regression fit is through the analysis of variance (ANOVA) decomposition, which partitions the total variability in the response into a component explained by the regression and a residual component. The F-statistic derived from this partition tests the overall significance of the linear relationship: H₀: β₁ = 0 versus H₁: β₁ ≠ 0. In simple linear regression, this F-test is algebraically equivalent to the two-sided t-test on b₁, but the ANOVA framework generalizes seamlessly to multiple regression.

The total sum of squares (SS_tot) is split into regression (SS_reg) and residual (SS_res) components. The F-statistic tests whether the regression component is large enough relative to the residual to reject H₀: β₁ = 0.

After fitting the model, examining the residuals is essential for validating assumptions. A residual-versus-fitted-values plot should reveal a random horizontal band centered at zero. Patterns such as curvature suggest non-linearity, and a funnel shape indicates heteroscedasticity. A normal Q–Q plot of the residuals checks the normality assumption; systematic departure from the diagonal line signals that parametric confidence intervals and p-values may not be trustworthy. In biostatistical applications—where sample sizes can be small (e.g., a pilot clinical trial with 20 patients)—these diagnostic plots should be examined before interpreting any regression output.

SECTION 6

Worked Example — Blood Pressure and BMI

Suppose a researcher collects data on n = 8 adults, measuring each subject's body mass index (BMI, in kg/m²) and resting systolic blood pressure (SBP, in mmHg). The research question is: Is there a significant linear association between BMI and systolic blood pressure, and if so, how much does SBP change on average for each one-unit increase in BMI?

Subject	BMI (X)	SBP (Y)
1	20	110
2	22	114
3	24	120
4	26	119
5	28	128
6	30	130
7	32	132
8	34	138

Dataset: BMI and systolic blood pressure for 8 adults

Fitting and Interpreting a Simple Linear Regression of SBP on BMI

1
Step 1 — Compute Summary Statistics

Calculate the sample means: x̄ = (20 + 22 + 24 + 26 + 28 + 30 + 32 + 34) / 8 = 216 / 8 = 27.0; ȳ = (110 + 114 + 120 + 119 + 128 + 130 + 132 + 138) / 8 = 991 / 8 = 123.875.

x̄ = 27.0, ȳ = 123.875
2
Step 2 — Compute S_xy and S_xx

S_xx = Σ(xᵢ − x̄)² = (−7)² + (−5)² + (−3)² + (−1)² + (1)² + (3)² + (5)² + (7)² = 49 + 25 + 9 + 1 + 1 + 9 + 25 + 49 = 168. S_xy = Σ(xᵢ − x̄)(yᵢ − ȳ) = (−7)(−13.875) + (−5)(−9.875) + (−3)(−3.875) + (−1)(−4.875) + (1)(4.125) + (3)(6.125) + (5)(8.125) + (7)(14.125) = 97.125 + 49.375 + 11.625 + 4.875 + 4.125 + 18.375 + 40.625 + 98.875 = 325.0.

S_xx = 168, S_xy = 325.0
3
Step 3 — Compute the Slope and Intercept

b₁ = S_xy / S_xx = 325.0 / 168 ≈ 1.935. b₀ = ȳ − b₁ × x̄ = 123.875 − 1.935 × 27.0 ≈ 123.875 − 52.245 ≈ 71.63. The fitted equation is ŷ = 71.63 + 1.935x.

ŷ = 71.63 + 1.935 × BMI
4
Step 4 — Interpret the Slope in Context

For each additional 1 kg/m² increase in BMI, systolic blood pressure is estimated to increase by approximately 1.94 mmHg on average. This is a conditional-mean statement: individual subjects will scatter around this average trend.
5
Step 5 — Compute R² and Assess Fit

SS_tot = Σ(yᵢ − ȳ)² = 13.875² + 9.875² + 3.875² + 4.875² + 4.125² + 6.125² + 8.125² + 14.125² ≈ 192.52 + 97.52 + 15.02 + 23.77 + 17.02 + 37.52 + 66.02 + 199.52 ≈ 648.875. SS_reg = b₁² × S_xx = 1.935² × 168 ≈ 3.744 × 168 ≈ 628.99. R² = 628.99 / 648.875 ≈ 0.969. Approximately 96.9% of the variability in systolic blood pressure in this sample is explained by BMI.

R² ≈ 0.969 — very strong linear association
6
Step 6 — Test H₀: β₁ = 0

SS_res = SS_tot − SS_reg = 648.875 − 628.99 ≈ 19.89. MS_res = 19.89 / (8 − 2) = 3.315. SE(b₁) = √(3.315 / 168) ≈ √0.01973 ≈ 0.1405. t = b₁ / SE(b₁) = 1.935 / 0.1405 ≈ 13.77 on 6 df. The critical value at α = 0.05 (two-tailed) is ≈ 2.447, so we soundly reject H₀ (p < 0.001). There is strong evidence that BMI is linearly associated with systolic blood pressure in this sample.

t = 13.77, df = 6, p < 0.001 → Reject H₀

SECTION 7

Strengths, Limitations & Common Pitfalls

Simple linear regression is among the most widely used analytic tools in biostatistics, but its simplicity is both its greatest virtue and its most significant constraint. The table below summarizes the key strengths and limitations a researcher should weigh before relying on a simple linear regression analysis.

Strengths	Limitations
Highly interpretable: slope directly expresses the change in Y per unit change in X.	Restricted to one predictor; cannot control for confounders.
Closed-form estimators (no iterative algorithm needed), making computation fast and transparent.	Assumes a strictly linear mean function; curvilinear relationships are missed or mischaracterized.
Provides a full inferential framework: confidence intervals, hypothesis tests, prediction intervals.	Sensitive to outliers and influential points, especially in small samples.
R² offers an intuitive measure of explained variability.	R² can be misleadingly high or low depending on the range of X sampled.
Serves as the building block for multiple regression, ANCOVA, and generalized linear models.	Extrapolation beyond the observed range of X is unreliable and potentially dangerous in clinical contexts.

Strengths vs. limitations of simple linear regression in biostatistics

⚠

COMMON PITFALL — CORRELATION ≠ CAUSATION

A statistically significant slope (p < 0.05) demonstrates association, not causation. In observational biostatistical studies, unmeasured confounders may drive the apparent relationship. For example, a regression of ice-cream sales on drowning rates produces a significant positive slope, but both variables are confounded by temperature. Always interpret regression output in the context of the study design: randomized experiments support causal claims far more convincingly than observational cross-sections.

SECTION 8

Connection to Multiple Regression & Advanced Modeling

Simple linear regression is the gateway to a family of increasingly flexible models that address the limitations of a single-predictor, straight-line fit. In practice, biostatisticians rarely stop at one predictor: multiple linear regression adds additional covariates to adjust for confounding, while generalized linear models (GLMs) extend the framework to non-normal outcomes such as binary events (logistic regression) or counts (Poisson regression). The table below highlights the key differences between simple linear regression and its immediate extensions.

Feature	Simple Linear Regression	Multiple Linear Regression	GLM (e.g., Logistic)
Number of predictors	1	≥ 2	≥ 1
Response type	Continuous, normal	Continuous, normal	Binary, count, etc.
Link function	Identity (ŷ = b₀ + b₁x)	Identity	Logit, log, etc.
Estimation	Closed-form OLS	Closed-form OLS	Iteratively reweighted LS / MLE
Confounding control	None	Yes, via covariates	Yes, via covariates

Simple linear regression in the broader modeling ecosystem

Despite these extensions, the interpretive logic of simple linear regression carries forward: every coefficient in a multiple regression or GLM still represents the change in the (transformed) expected response per one-unit change in the corresponding predictor, holding all other predictors constant. Mastering the one-predictor case therefore provides the conceptual scaffolding for every regression model you will encounter in biostatistics.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

Explain, in your own words, why the ordinary least-squares criterion squares the residuals rather than simply summing the raw (signed) residuals. What problem would arise if we minimized Σeᵢ instead of Σeᵢ²?

Show answer

Summing the raw residuals Σeᵢ would always equal zero for any line passing through (x̄, ȳ), making the criterion useless for distinguishing among candidate lines. By squaring the residuals, positive and negative deviations no longer cancel, and large deviations are penalized more heavily than small ones. The sum of squared residuals therefore provides a unique, strictly positive measure of overall misfit that has a single global minimum—the OLS line. Additionally, the squared loss function leads to elegant closed-form estimators and connects naturally to the normal distribution's maximum-likelihood estimation.

Question 2: BASIC CALCULATION

A study of n = 5 laboratory mice yields x̄ = 10 (drug dose in mg), ȳ = 50 (tumor volume in mm³), S_xx = 40, and S_xy = −120. Compute the slope b₁, the intercept b₀, and write the fitted regression equation. Interpret the slope in context.

Show answer

b₁ = S_xy / S_xx = −120 / 40 = −3.0. b₀ = ȳ − b₁ × x̄ = 50 − (−3.0)(10) = 50 + 30 = 80. The fitted equation is ŷ = 80 − 3.0x. Interpretation: for each additional 1 mg increase in drug dose, the average tumor volume is estimated to decrease by 3.0 mm³. The negative slope indicates an inverse association between dose and tumor volume in this sample, which is biologically consistent with an effective anti-tumor drug. The intercept of 80 mm³ represents the predicted average tumor volume when the dose is zero (the control condition).

Question 3: INTERMEDIATE

Using the data from Problem 2, suppose SS_tot = 400. Compute SS_reg, SS_res, R², the residual mean square MS_res, and the standard error of b₁. Then compute the t-statistic for testing H₀: β₁ = 0 and state whether you reject H₀ at α = 0.05 (critical t with 3 df ≈ 3.182).

Show answer

SS_reg = b₁² × S_xx = (−3)² × 40 = 9 × 40 = 360. SS_res = SS_tot − SS_reg = 400 − 360 = 40. R² = 360 / 400 = 0.90 (90% of the variability in tumor volume is explained by drug dose). MS_res = SS_res / (n − 2) = 40 / 3 ≈ 13.33. SE(b₁) = √(MS_res / S_xx) = √(13.33 / 40) = √0.3333 ≈ 0.577. t = b₁ / SE(b₁) = −3.0 / 0.577 ≈ −5.20. Since |−5.20| = 5.20 > 3.182, we reject H₀ at α = 0.05. There is statistically significant evidence of a linear association between drug dose and tumor volume.

Question 4: APPLIED

An epidemiologist fits a simple linear regression of fasting blood glucose (mg/dL) on waist circumference (cm) using data from 150 participants and obtains: ŷ = 22.4 + 1.12x, R² = 0.35, SE(b₁) = 0.14, p < 0.001. (a) Interpret the slope in a sentence suitable for a clinical audience. (b) Construct an approximate 95% confidence interval for β₁. (c) A colleague claims that waist circumference 'explains most of the variation in blood glucose.' Evaluate this claim using R².

Show answer

(a) For each additional centimeter of waist circumference, fasting blood glucose is estimated to increase by 1.12 mg/dL on average in this population—a clinically meaningful increment given that the cumulative effect over a 10 cm difference would be roughly 11 mg/dL. (b) With n = 150, df = 148, and t* ≈ 1.976, the 95% CI is 1.12 ± 1.976 × 0.14 = 1.12 ± 0.277, giving (0.843, 1.397). Since this interval excludes zero, the association is statistically significant. (c) R² = 0.35 means waist circumference explains only 35% of the variability in blood glucose; 65% remains unexplained. The colleague's claim is not supported—while the relationship is statistically significant, waist circumference alone does not explain 'most' of the variation. Other factors such as diet, genetics, physical activity, and insulin resistance likely account for the unexplained variance.

Question 5: CRITICAL THINKING

A researcher regresses resting heart rate (bpm) on age (years) using data from 200 healthy adults aged 25–65 and obtains b₁ = 0.18 (SE = 0.06, p = 0.003). She then uses the model to predict resting heart rate for an 85-year-old patient. (a) Discuss the validity of this prediction. (b) The researcher next notices a clear U-shaped pattern in the residual-versus-fitted-values plot. What does this imply about the model, and what remedial action might she consider? (c) If the study were observational, could she conclude that aging causes heart rate to increase? Why or why not?

Show answer

(a) Predicting for an 85-year-old constitutes extrapolation, because the model was fit on ages 25–65. The linear relationship may not hold outside that range—heart rate dynamics in the elderly may plateau or even decline due to medication use or autonomic changes. The prediction is therefore unreliable and potentially misleading for clinical decision-making. (b) A U-shaped (quadratic) pattern in the residual plot indicates that the linearity assumption is violated: the true relationship between age and heart rate is curvilinear. Remedial options include adding a quadratic term (age²) to the model—moving to polynomial regression—or applying a transformation (e.g., log) to the predictor or response. (c) No. An observational study cannot establish causation because of the potential for confounding variables (e.g., changes in physical activity, body composition, or medication use with age) and reverse causality. Only a randomized experiment—impossible here since age cannot be assigned—would support a causal claim. The researcher should frame her conclusion as 'age is positively associated with resting heart rate after accounting for the limitations of an observational design.'

SUMMARY

Lesson Summary

Summary

Simple linear regression models the relationship between a single continuous predictor X and a continuous response Y using the equation Yᵢ = β₀ + β₁Xᵢ + εᵢ. The ordinary least-squares method estimates the slope b₁ = S_xy / S_xx and intercept b₀ = ȳ − b₁x̄ by minimizing the sum of squared residuals. Four key assumptions—linearity, independence, normality of residuals, and homoscedasticity—must be checked via residual diagnostics before interpreting inference.

The slope b₁ is interpreted as the estimated average change in Y per one-unit increase in X—always stated in the units of the specific problem. The coefficient of determination R² quantifies explained variability, while the t-test on b₁ (or equivalently the ANOVA F-test) evaluates whether the linear association is statistically significant. Remember that significance does not imply causation in observational studies, and extrapolation beyond the observed range of X should be avoided. Simple linear regression is the foundational model upon which multiple regression and generalized linear models are built, making fluency with its mechanics and interpretation essential for any biostatistical analysis.