Outliers & Influential Points — I can identify outliers and influential points and describe how they may affect a model. - Math 2

SECTION 1

Historical Context & Motivation

For centuries, scientists and mathematicians collecting data have faced the same nagging question: what do you do with measurements that don't seem to belong? A single strange data point can sometimes reveal an exciting new discovery, but it can also be the result of a simple mistake. The history of statistics is full of moments when researchers had to decide whether to trust an unusual observation or set it aside.

As early as the 1800s, astronomers struggled with this problem while mapping the positions of stars. One faulty telescope reading could throw off an entire star chart. Over time, mathematicians developed formal methods to identify and handle these unusual values — what we now call outliers and influential points. Understanding these concepts is essential whenever you build a model from data.

1805 • Legendre's Least Squares

Adrien-Marie Legendre publishes the method of least squares, giving mathematicians a way to fit lines to data — and making them vulnerable to outliers pulling the line off course.

1886 • Galton & Regression

Francis Galton introduces the idea of regression toward the mean, showing that extreme observations tend to be less extreme on re-measurement — hinting at the importance of spotting unusual data.

1969 • Anscombe's Quartet

Frank Anscombe constructs four datasets with identical summary statistics but wildly different scatter plots, dramatically illustrating how a single outlier can distort a linear model.

1977 • Cook's Distance

R. Dennis Cook introduces Cook's distance, a formal measure of how much each data point influences a regression line — turning the intuitive idea of 'influential point' into a precise calculation.

The core question these developments address is straightforward: when you fit a line (or curve) to data, which points have an outsized effect on the result, and what should you do about them? Answering this question carefully is what separates a trustworthy model from a misleading one.

SECTION 2

Core Principles & Definitions

Before diving into calculations, you need a clear vocabulary. Three closely related but distinct ideas sit at the heart of this topic. While people sometimes use the words interchangeably, each one describes a different relationship between a data point and the model.

Outlier

A data point whose y-value (response) is far from the value the model predicts. It has a large residual. An outlier does not necessarily change the slope or intercept by much.

High-Leverage Point

A data point whose x-value is far from the mean of the other x-values. Because it sits on the edge of the data, it has the potential to pull the regression line toward itself.

Influential Point

A point is influential if removing it causes a noticeable change in the regression equation (slope, intercept, or r²). Influential points are usually high-leverage points that are also outliers — a dangerous combination.

Residual

The vertical distance between an observed y-value and the predicted ŷ on the regression line: residual = y − ŷ. Large absolute residuals flag possible outliers.

✦

KEY TAKEAWAY

Think of a regression line as a seesaw balanced on the mean of the data. A point sitting far out along the seesaw arm (high leverage) can tilt the whole seesaw, especially if it is also far above or below the board (outlier). When it both sits far out and is far off the trend, it's an influential point — it can tip the whole model.

SECTION 3

Visual Explanation

The scatter plot below shows a small dataset with a clear linear trend. Three special points are highlighted: one that is merely an outlier (large residual but near the center of x-values), one that has high leverage but follows the trend, and one that is both an outlier and a high-leverage point — making it influential.

Point A is an outlier — it has a large residual but sits near the center of the x-values, so it has little leverage. Point B has high leverage because its x-value is far from the mean, but it falls right on the trend, so it doesn't distort the line. Point C is an influential point — it has extreme x and a large residual, pulling the regression line significantly if included.

Notice the dashed vertical lines on points A and C — these represent the residuals. Point A's residual is large, but because A sits near the center of the x-values, it doesn't have much pull on where the line pivots. Point C's residual is also large, and it sits at the far edge of the data. That combination makes C influential: it can rotate the line toward itself and change the slope, intercept, and correlation coefficient substantially.

SECTION 4

Mathematical Framework

You already know that a least-squares regression line minimizes the sum of squared residuals. The key formulas below show exactly how each data point feeds into the slope and intercept — and why extreme values can hijack the result.

RESIDUAL

eᵢ = yᵢ − ŷᵢ

where yᵢ is the observed value and ŷᵢ is the value predicted by the regression line. A large |eᵢ| signals a potential outlier.

SLOPE OF REGRESSION LINE

b₁ = r × (s_y / s_x)

Here r is the correlation coefficient, $s_y$ is the standard deviation of y, and $s_x$ is the standard deviation of x. An influential point can change r, $s_x$ , and $s_y$ simultaneously.

INTERCEPT

b₀ = ȳ − b₁ × x̄

The regression line always passes through (x̄, ȳ). An influential point shifts the means x̄ and ȳ, which shifts the intercept and, through b₁, the slope.

The practical test for influence is simple: fit the regression line with all points, then fit it again with the suspect point removed. If the slope, intercept, or r² change noticeably, the point is influential. In more advanced courses, this idea is formalized through Cook's distance, but for now the remove-and-compare approach is the standard method you'll use.

SECTION 5

Classifying Unusual Points

The diagram below organizes every data point into one of four categories based on two questions: Is the x-value extreme (high leverage)? Is the residual large (outlier)? The most dangerous combination — extreme x and large residual — is the influential point.

This 2 × 2 classification grid shows how leverage (extreme x) and residual size combine. Only the bottom-right quadrant — influential points — can substantially distort a regression model.

Keep in mind that a high-leverage point that falls on the existing trend actually strengthens the model — it increases r² and makes the slope estimate more precise. Leverage is not automatically bad. It only becomes a problem when paired with a large residual, because then the point is tugging the line away from the overall pattern.

💡

Quick Identification Checklist

To classify any suspect point: (1) Check if its x-value is far from x̄ — if yes, it has high leverage. (2) Check if its residual is unusually large — if yes, it's an outlier. (3) Remove the point and re-fit the line — if the equation changes substantially, the point is influential.

SECTION 6

Worked Example

Suppose you are studying the relationship between hours of study (x) and exam score (y) for ten students. After plotting the data, you notice one student studied 12 hours — much more than anyone else — and scored 58, which is well below the predicted score for that many hours. Let's investigate whether this point is influential.

Student	Hours (x)	Score (y)
1	1	52
2	2	58
3	3	62
4	3	67
5	4	70
6	4	73
7	5	75
8	5	78
9	6	82
10	12	58

Study hours vs. exam score for 10 students

Is Student 10 an Influential Point?

1
Step 1 — Check for High Leverage

Calculate the mean of x for all ten students: x̄ = (1 + 2 + 3 + 3 + 4 + 4 + 5 + 5 + 6 + 12) / 10 = 45 / 10 = 4.5. Student 10's x-value of 12 is far above x̄ = 4.5, so this point has high leverage.

x̄ = 4.5; Student 10's x = 12 → High leverage ✓
2
Step 2 — Fit the Regression Line (All 10 Points)

Using a calculator or statistical software, the regression line with all 10 points is approximately ŷ = 52.7 + 2.8x, with r² ≈ 0.42. This relatively low r² already hints that something may be pulling the model off track.

ŷ = 52.7 + 2.8x, r² ≈ 0.42
3
Step 3 — Compute the Residual for Student 10

Predicted score: ŷ = 52.7 + 2.8(12) = 52.7 + 33.6 = 86.3. Actual score: y = 58. Residual: 58 − 86.3 = −28.3. This is a very large negative residual, confirming Student 10 is also an outlier.

Residual = −28.3 → Outlier ✓
4
Step 4 — Remove the Point and Re-Fit

Removing Student 10 and fitting a line to the remaining 9 students gives approximately ŷ = 45.8 + 5.9x, with r² ≈ 0.97. The slope nearly doubled (from 2.8 to 5.9), the intercept changed noticeably, and r² jumped from 0.42 to 0.97. That is a dramatic change.

Without Student 10: ŷ = 45.8 + 5.9x, r² ≈ 0.97
5
Step 5 — Conclusion

Student 10 has high leverage (extreme x) and a large residual (outlier). Removing it drastically changes the slope, intercept, and r². Therefore, Student 10 is an influential point. You would want to investigate — did this student really study 12 hours, or was the data entered incorrectly? If the data is valid, reporting both models (with and without the point) is good practice.

Student 10 is an influential point.

SECTION 7

How Outliers & Influential Points Affect a Model

Not every unusual point is equally dangerous. The table below compares how different types of unusual points affect the three most important summary measures of a linear model: the slope, the intercept, and the coefficient of determination (r²).

Type of Point	Effect on Slope (b₁)	Effect on r²	Should You Worry?
Outlier (central x)	Minimal change in slope because leverage is low.	May decrease r² because it adds unexplained variation.	Moderate — investigate the data value, but the model is fairly stable.
High-leverage point (on trend)	Slope stays roughly the same; may stabilize the estimate.	Often increases r² — extends the range of x, reinforcing the pattern.	Low — this point actually helps the model, but verify the data.
Influential point (high leverage + outlier)	Can drastically change the slope — may flatten, steepen, or even reverse the line.	Can either inflate or deflate r² unpredictably.	High — always investigate. Report results with and without the point.

Comparison of effects on the regression model

✦

KEY TAKEAWAY

Imagine you're taking a class photo and everyone is standing in a straight line — that's your regression. An outlier is like someone jumping way above the group; it looks strange but doesn't move the line much. A high-leverage person at the far end who stays in line actually anchors the photo. But if that person at the far end also jumps way up, the whole line tilts — that's an influential point.

SECTION 8

Connection to Advanced Theory

In this course, you identify influential points by the remove-and-compare method. In more advanced statistics courses — including AP Statistics and college-level regression analysis — formal numerical diagnostics are used to automate and quantify this process.

Concept in This Course	Advanced Version
Visually checking if x is far from x̄	Leverage statistic (hᵢ) — a number between 0 and 1 measuring exactly how far xᵢ is from the data center.
Checking if the residual is large	Standardized / Studentized residuals — residuals divided by their estimated standard error, so \|e*\| > 2 flags an outlier.
Removing a point and re-fitting	Cook's distance (Dᵢ) — combines leverage and residual size into a single measure of influence. Dᵢ > 0.5 is often flagged.
Describing effect on slope / r²	DFFITS and DFBETAS — measure how much the fitted value or individual coefficient changes when a point is deleted.

From intuition to formal diagnostics

You don't need to memorize these advanced formulas now, but it's useful to know they exist. The intuition you're building — that extreme x-values combined with large residuals spell trouble — is exactly the same intuition that underlies every one of these formal tools. Mastering the visual and remove-and-compare approach now will make the transition to these numerical diagnostics straightforward later.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

Explain the difference between an outlier and an influential point. Can a point be one without being the other? Give an example situation for each case.

Show answer

An outlier is a point with a large residual — its y-value is far from what the model predicts. An influential point is one whose removal significantly changes the regression equation. A point can be an outlier without being influential if its x-value is near the mean (low leverage), because it lacks the 'arm length' to tilt the line. Conversely, a high-leverage point on the trend line is not an outlier at all. A point is typically influential when it is both an outlier and a high-leverage point — extreme in x and far from the predicted y.

Question 2: BASIC CALCULATION

A regression line is ŷ = 10 + 3x. A data point has coordinates (5, 30). Calculate the residual and state whether the point is above or below the line.

Show answer

First, find the predicted value: ŷ = 10 + 3(5) = 10 + 15 = 25. The residual is y − ŷ = 30 − 25 = 5. Since the residual is positive, the actual point is 5 units above the regression line. Whether this residual is 'large' enough to call the point an outlier depends on the spread of the other residuals, but a residual of +5 is definitely worth investigating if most residuals are close to zero.

Question 3: INTERMEDIATE

A dataset of 8 points has x̄ = 6. With all 8 points, the regression equation is ŷ = 20 + 4x (r² = 0.91). When a point at (15, 35) is removed, the equation becomes ŷ = 18 + 4.8x (r² = 0.95). Classify the point at (15, 35): is it a high-leverage point, an outlier, an influential point, or some combination? Justify your answer.

Show answer

The point has x = 15, which is far from x̄ = 6, so it has high leverage. Its predicted value with the full model is ŷ = 20 + 4(15) = 80, but its actual y is 35, giving a residual of 35 − 80 = −45 — a huge residual, making it an outlier. Removing the point changes the slope from 4.0 to 4.8 (a 20% increase) and r² from 0.91 to 0.95, so the point is influential. It qualifies as all three: high-leverage, outlier, and influential. This makes sense because influence is the combined effect of high leverage and outlier status.

Question 4: APPLIED

A car dealership records the age (years) and price ($) of 20 used cars. The regression model is ŷ = 28,000 − 2,100x with r² = 0.87. One car is 15 years old and priced at $22,000. The other 19 cars range from 1 to 8 years old. When the 15-year-old car is removed, the model becomes ŷ = 30,500 − 2,800x with r² = 0.93. Explain what happened and what you would recommend to the analyst.

Show answer

The 15-year-old car has very high leverage because its age (x = 15) is far beyond the range of the other cars (1–8 years). Its predicted price is ŷ = 28,000 − 2,100(15) = −$3,500, which is nonsensical, while its actual price is $22,000, producing a massive positive residual. This point is pulling the slope toward zero (flattening the line) and lowering r². Removing it increases the slope magnitude from 2,100 to 2,800 and r² from 0.87 to 0.93, confirming it is highly influential. The analyst should investigate whether a 15-year-old car is an appropriate part of this analysis — it may represent a different market (classic or specialty car). Reporting both models and noting the influence of this point would be the most transparent approach.

Question 5: CRITICAL THINKING

A student argues: 'We should always remove influential points because they make the model worse.' Do you agree or disagree? Write a short paragraph defending your position, using at least two specific reasons.

Show answer

Disagree. While influential points can distort a model, automatically removing them is not good practice for at least two reasons. First, the influential point may represent legitimate data — for example, a CEO's salary in a dataset of company employees is extreme but real, and removing it hides important information about the population. Second, the presence of an influential point may reveal that a linear model is the wrong choice; perhaps the true relationship is curved, and removing the point masks this fact. Instead of deleting, you should investigate the cause of the unusual value (data entry error, measurement issue, or genuine variation), report results both with and without the point, and consider whether a different model better fits the full dataset. Removing data without justification can bias your conclusions and undermine the credibility of your analysis.

SUMMARY

Lesson Summary

Summary

An outlier is a data point with a large residual — its observed y-value is far from the value predicted by the regression line. A high-leverage point has an x-value far from the mean of x, giving it the potential to swing the line. When a point is both an outlier and a high-leverage point, it becomes an influential point — removing it causes a noticeable change in the slope, intercept, or r² of the regression model.

To identify these points, compare the regression equation with all data included to the equation after removing the suspect point. If the model changes substantially, the point is influential. Always investigate why a point is unusual — it might be a data error, a measurement from a different population, or a genuine extreme value — before deciding whether to include it in your analysis. Reporting results both with and without the influential point is the most responsible practice.