Historical Context & Motivation
Humans have always tried to find patterns in data. Long before computers, astronomers and mathematicians sought ways to draw the "best" curve through measured observations. The idea of regression — fitting a mathematical function to data points — grew from the practical need to predict planetary orbits, population growth, and economic trends. Over centuries, this single idea evolved into one of the most powerful tools in modern statistics.
The central question this topic addresses is straightforward: when you have a set of data, which type of function — linear, quadratic, cubic, exponential, power, or sinusoidal — best describes the pattern? And how do you measure "best"? The answers lie in residual analysis and goodness-of-fit statistics.
Core Principles & Definitions
Before diving into calculations, you need a clear understanding of the key ideas that underpin regression and model comparison. These concepts work together like a toolkit: the regression model provides a prediction, residuals tell you how far off those predictions are, and R² summarizes the overall quality of the fit.
Regression Model
Residual
Coefficient of Determination (R²)
Sum of Squared Residuals (SSres)
Model Comparison
Visual Explanation — Residuals on a Scatter Plot
The diagram below shows six data points with a fitted linear regression line. The vertical segments between each point and the line represent the residuals. Notice that some segments point upward (positive residuals) and some point downward (negative residuals). In a good model, these residuals should appear randomly scattered — no obvious pattern. If the residuals show a curve, a non-linear model may be a better choice.
In the diagram, notice that the two leftmost points sit below the line, the middle points sit above, and the rightmost point is close to the line. This pattern — negative, then positive, then negative — hints that the data may curve, and a quadratic model might yield smaller residuals and a higher R².
Mathematical Framework
The mathematics behind regression and model comparison rests on a few core formulas. Your GDC (graphing display calculator) handles the heavy computation, but understanding what the formulas mean helps you interpret results and explain your reasoning on IB exams.
Comparing Different Regression Models
In IB AI HL, you are expected to use your GDC to fit several types of models to the same dataset and then decide which fits best. The table below summarizes the regression types you should know, along with their general shapes and typical applications.
| Model Type | Equation Form | Typical Shape | When to Use |
|---|---|---|---|
| Linear | y = ax + b | Straight line | Constant rate of change |
| Quadratic | y = ax² + bx + c | Parabola (U-shape) | Data rises then falls (or vice versa) |
| Cubic | y = ax³ + bx² + cx + d | S-curve or wave | Two turning points visible in data |
| Exponential | y = a × bˣ + c | Rapid growth or decay | Population growth, radioactive decay |
| Power | y = axⁿ | Curve through origin | Scaling laws, physics relationships |
| Sinusoidal | y = a sin(bx + c) + d | Oscillating wave | Seasonal or periodic data |
When comparing models, don't just pick the highest R². Always check the residual plot for patterns. A model with slightly lower R² but randomly scattered residuals is often more reliable than one with higher R² but patterned residuals, because patterns suggest the model is systematically missing something in the data's behavior.
Worked Example — Choosing the Best Model
A biologist measures the mass of a bacterial colony (in mg) over several hours. The data is shown below. We will fit a linear and an exponential model, calculate residuals, and compare R² values.
| Time (hours) | Mass (mg) |
|---|---|
| 0 | 2.0 |
| 1 | 2.8 |
| 2 | 4.2 |
| 3 | 5.9 |
| 4 | 8.5 |
| 5 | 12.1 |
Interpolation vs. Extrapolation
Once you have fitted a regression model, you can use it to make predictions. However, where you make those predictions matters enormously. The IB AI HL specification explicitly requires you to understand the distinction between interpolation and extrapolation, and this distinction is frequently assessed on IB papers.
Interpolation
Extrapolation
| Feature | Interpolation | Extrapolation |
|---|---|---|
| x-value location | Within the range of observed data | Outside the range of observed data |
| Reliability | Generally reliable — the model has been validated across the surrounding region | Unreliable — the pattern established by the data may not continue |
| Risk level | Low | High — the further outside the data range, the greater the uncertainty |
| IB requirement | Acceptable to use the model for predictions | Must acknowledge limitations; state that prediction may not be valid |
| Example | Data for t = 0 to 10; predicting at t = 6 | Data for t = 0 to 10; predicting at t = 25 |
Consider a concrete example: a model fitted to bacterial growth data for hours t = 0 to 5 (as in the worked example) gives y = 1.95 × 1.39ˣ. Predicting the mass at t = 3 (interpolation) is reliable. Predicting the mass at t = 20 (extrapolation) is not, because the exponential model would suggest an unrealistically large colony mass, and real bacterial growth is eventually limited by resources. The mathematical model does not know about those real-world constraints — only the analyst does.
Strengths & Limitations of R² and Residual Analysis
R² and residual analysis are powerful tools, but like any tool, they have boundaries. Understanding both their strengths and limitations will help you avoid common pitfalls on exams and in real-world applications.
| Strengths | Limitations |
|---|---|
| R² gives a single, easy-to-interpret number for model quality (0 to 1). | R² always increases (or stays the same) when you add more parameters, even if the extra complexity isn't justified. |
| Residual plots reveal systematic patterns that a single number like R² might miss. | Residual analysis requires visual judgment, which can be subjective with small datasets. |
| Can compare completely different model types (e.g., exponential vs. quadratic) on the same data. | A high R² doesn't mean the model is correct — it only means the model fits the observed data well. Extrapolation is risky. |
| Identifying a good model allows reliable interpolation within the data range. | Outliers can heavily distort both R² and the fitted model, especially with small samples. |
Connection to Advanced Theory
The regression and model comparison skills you learn in IB AI HL form the foundation for more sophisticated statistical methods used in university courses and professional data science. The table below shows how the concepts you've mastered connect to what comes next.
| IB AI HL Concept | Advanced Extension |
|---|---|
| R² (coefficient of determination) | Adjusted R², which penalizes models for having too many parameters — addresses the limitation of R² always increasing. |
| Visual residual analysis | Formal hypothesis tests on residuals (e.g., Durbin-Watson test for autocorrelation, Shapiro-Wilk test for normality). |
| Comparing 2–3 model types | Information criteria (AIC, BIC) that balance model fit against complexity, allowing comparison of dozens of models. |
| Single independent variable (x) | Multiple regression with several independent variables (x₁, x₂, x₃, …) — used heavily in economics, medicine, and machine learning. |
If you continue into statistics, data science, or any science that relies on modeling, the intuition you build now — checking residuals, comparing R² values, and thinking critically about whether a model makes contextual sense — will serve you in every course. The core logic never changes, even as the tools grow more sophisticated.
Practice Problems
Lesson Summary
In this lesson you learned that regression is the process of fitting a mathematical function — linear, quadratic, cubic, exponential, power, or sinusoidal — to a set of data points. A residual (e = y − ŷ) measures the vertical distance between each observed value and the model's prediction. By squaring and summing all residuals, you get SSres, and comparing it to the total variation SStot gives you the coefficient of determination R² = 1 − (SSres / SStot).
To choose the best model, fit multiple regression types to the same data using your GDC, then compare their R² values — higher is better — and examine residual plots for patterns. A good model has a high R² and randomly scattered residuals. When using a model to make predictions, always distinguish between interpolation (predicting within the data range — reliable) and extrapolation (predicting outside the data range — unreliable). Always consider real-world context — does the model type make sense for the situation? — and beware of overfitting, where an overly complex model fits the noise rather than the true pattern.