Outliers & Influential Points — I can identify outliers and...

SECTION 1

Historical Context & Motivation

For centuries, scientists and mathematicians collecting data have faced the same nagging question: what do you do with measurements that don't seem to belong? A single strange data point can sometimes reveal an exciting new discovery, but it can also be the result of a simple mistake. The history of statistics is full of moments when researchers had to decide whether to trust an unusual observation or set it aside.

As early as the 1800s, astronomers struggled with this problem while mapping the positions of stars. One faulty telescope reading could throw off an entire star chart. Over time, mathematicians developed formal methods to identify and handle these unusual values — what we now call outliers and influential points. Understanding these concepts is essential whenever you build a model from data.

1805

Legendre's Least Squares

Adrien-Marie Legendre publishes the method of least squares, giving mathematicians a way to fit lines to data — and making them vulnerable to outliers pulling the line off course.

1886

Galton & Regression

Francis Galton introduces the idea of regression toward the mean, showing that extreme observations tend to be less extreme on re-measurement — hinting at the importance of spotting unusual data.

1969

Anscombe's Quartet

Frank Anscombe constructs four datasets with identical summary statistics but wildly different scatter plots, dramatically illustrating how a single outlier can distort a linear model.

1977

Cook's Distance

R. Dennis Cook introduces Cook's distance, a formal measure of how much each data point influences a regression line — turning the intuitive idea of 'influential point' into a precise calculation.

The core question these developments address is straightforward: when you fit a line (or curve) to data, which points have an outsized effect on the result, and what should you do about them? Answering this question carefully is what separates a trustworthy model from a misleading one.

SECTION 2

Core Principles & Definitions

Before diving into calculations, you need a clear vocabulary. Three closely related but distinct ideas sit at the heart of this topic. While people sometimes use the words interchangeably, each one describes a different relationship between a data point and the model.

Outlier

A data point whose y-value (response) is far from the value the model predicts. It has a large residual. An outlier does not necessarily change the slope or intercept by much.

High-Leverage Point

A data point whose x-value is far from the mean of the other x-values. Because it sits on the edge of the data, it has the potential to pull the regression line toward itself.

Influential Point

A point is influential if removing it causes a noticeable change in the regression equation (slope, intercept, or r²). Influential points are usually high-leverage points that are also outliers — a dangerous combination.

Residual

The vertical distance between an observed y-value and the predicted ŷ on the regression line: residual = y − ŷ. Large absolute residuals flag possible outliers.

✦ KEY TAKEAWAY

Think of a regression line as a seesaw balanced on the mean of the data. A point sitting far out along the seesaw arm (high leverage) can tilt the whole seesaw, especially if it is also far above or below the board (outlier). When it both sits far out and is far off the trend, it's an influential point — it can tip the whole model.

SECTION 3

Visual Explanation

The scatter plot below shows a small dataset with a clear linear trend. Three special points are highlighted: one that is merely an outlier (large residual but near the center of x-values), one that has high leverage but follows the trend, and one that is both an outlier and a high-leverage point — making it influential.

Point A is an outlier — it has a large residual but sits near the center of the x-values, so it has little leverage. Point B has high leverage because its x-value is far from the mean, but it falls right on the trend, so it doesn't distort the line. Point C is an influential point — it has extreme x and a large residual, pulling the regression line significantly if included.

Notice the dashed vertical lines on points A and C — these represent the residuals. Point A's residual is large, but because A sits near the center of the x-values, it doesn't have much pull on where the line pivots. Point C's residual is also large, and it sits at the far edge of the data. That combination makes C influential: it can rotate the line toward itself and change the slope, intercept, and correlation coefficient substantially.

SECTION 4

Mathematical Framework

You already know that a least-squares regression line minimizes the sum of squared residuals. The key formulas below show exactly how each data point feeds into the slope and intercept — and why extreme values can hijack the result.

RESIDUAL

eᵢ = yᵢ − ŷᵢ

where yᵢ is the observed value and ŷᵢ is the value predicted by the regression line. A large |eᵢ| signals a potential outlier.

SLOPE OF REGRESSION LINE

b₁ = r × (s_y / s_x)

Here r is the correlation coefficient, s_y is the standard deviation of y, and s_x is the standard deviation of x. An influential point can change r, s_x, and s_y simultaneously.

INTERCEPT

b₀ = ȳ − b₁ × x̄

The regression line always passes through (x̄, ȳ). An influential point shifts the means x̄ and ȳ, which shifts the intercept and, through b₁, the slope.

The practical test for influence is simple: fit the regression line with all points, then fit it again with the suspect point removed. If the slope, intercept, or r² change noticeably, the point is influential. In more advanced courses, this idea is formalized through Cook's distance, but for now the remove-and-compare approach is the standard method you'll use.

SECTION 5

Classifying Unusual Points

The diagram below organizes every data point into one of four categories based on two questions: Is the x-value extreme (high leverage)? Is the residual large (outlier)? The most dangerous combination — extreme x and large residual — is the influential point.

This 2 × 2 classification grid shows how leverage (extreme x) and residual size combine. Only the bottom-right quadrant — influential points — can substantially distort a regression model.

Keep in mind that a high-leverage point that falls on the existing trend actually strengthens the model — it increases r² and makes the slope estimate more precise. Leverage is not automatically bad. It only becomes a problem when paired with a large residual, because then the point is tugging the line away from the overall pattern.

📋 Quick Identification Checklist

To classify any suspect point: (1) Check if its x-value is far from x̄ — if yes, it has high leverage. (2) Check if its residual is unusually large — if yes, it's an outlier. (3) Remove the point and re-fit the line — if the equation changes substantially, the point is influential.

SECTION 6

Worked Example

Suppose you are studying the relationship between hours of study (x) and exam score (y) for ten students. After plotting the data, you notice one student studied 12 hours — much more than anyone else — and scored 58, which is well below the predicted score for that many hours. Let's investigate whether this point is influential.

Study hours vs. exam score for 10 students

Student	Hours (x)	Score (y)
1	1	52
2	2	58
3	3	62
4	3	67
5	4	70
6	4	73
7	5	75
8	5	78
9	6	82
10	12	58

Is Student 10 an Influential Point?

Step 1 — Check for High LeverageCalculate the mean of x for all ten students: x̄ = (1 + 2 + 3 + 3 + 4 + 4 + 5 + 5 + 6 + 12) / 10 = 45 / 10 = 4.5. Student 10's x-value of 12 is far above x̄ = 4.5, so this point has high leverage.

x̄ = 4.5; Student 10's x = 12 → High leverage ✓

Step 2 — Fit the Regression Line (All 10 Points)Using a calculator or statistical software, the regression line with all 10 points is approximately ŷ = 52.7 + 2.8x, with r² ≈ 0.42. This relatively low r² already hints that something may be pulling the model off track.

ŷ = 52.7 + 2.8x, r² ≈ 0.42

Step 3 — Compute the Residual for Student 10Predicted score: ŷ = 52.7 + 2.8(12) = 52.7 + 33.6 = 86.3. Actual score: y = 58. Residual: 58 − 86.3 = −28.3. This is a very large negative residual, confirming Student 10 is also an outlier.

Residual = −28.3 → Outlier ✓

Step 4 — Remove the Point and Re-FitRemoving Student 10 and fitting a line to the remaining 9 students gives approximately ŷ = 45.8 + 5.9x, with r² ≈ 0.97. The slope nearly doubled (from 2.8 to 5.9), the intercept changed noticeably, and r² jumped from 0.42 to 0.97. That is a dramatic change.

Without Student 10: ŷ = 45.8 + 5.9x, r² ≈ 0.97

Step 5 — ConclusionStudent 10 has high leverage (extreme x) and a large residual (outlier). Removing it drastically changes the slope, intercept, and r². Therefore, Student 10 is an influential point. You would want to investigate — did this student really study 12 hours, or was the data entered incorrectly? If the data is valid, reporting both models (with and without the point) is good practice.

Student 10 is an influential point.

SECTION 7

How Outliers & Influential Points Affect a Model

Not every unusual point is equally dangerous. The table below compares how different types of unusual points affect the three most important summary measures of a linear model: the slope, the intercept, and the coefficient of determination (r²).

Comparison of effects on the regression model

Type of Point	Effect on Slope (b₁)	Effect on r²	Should You Worry?
Outlier (central x)	Minimal change in slope because leverage is low.	May decrease r² because it adds unexplained variation.	Moderate — investigate the data value, but the model is fairly stable.
High-leverage point (on trend)	Slope stays roughly the same; may stabilize the estimate.	Often increases r² — extends the range of x, reinforcing the pattern.	Low — this point actually helps the model, but verify the data.
Influential point (high leverage + outlier)	Can drastically change the slope — may flatten, steepen, or even reverse the line.	Can either inflate or deflate r² unpredictably.	High — always investigate. Report results with and without the point.

✦ KEY TAKEAWAY

Imagine you're taking a class photo and everyone is standing in a straight line — that's your regression. An outlier is like someone jumping way above the group; it looks strange but doesn't move the line much. A high-leverage person at the far end who stays in line actually anchors the photo. But if that person at the far end also jumps way up, the whole line tilts — that's an influential point.

SECTION 8

Connection to Advanced Theory

In this course, you identify influential points by the remove-and-compare method. In more advanced statistics courses — including AP Statistics and college-level regression analysis — formal numerical diagnostics are used to automate and quantify this process.

From intuition to formal diagnostics

Concept in This Course	Advanced Version
Visually checking if x is far from x̄	Leverage statistic (hᵢ) — a number between 0 and 1 measuring exactly how far xᵢ is from the data center.
Checking if the residual is large	Standardized / Studentized residuals — residuals divided by their estimated standard error, so \|e*\| > 2 flags an outlier.
Removing a point and re-fitting	Cook's distance (Dᵢ) — combines leverage and residual size into a single measure of influence. Dᵢ > 0.5 is often flagged.
Describing effect on slope / r²	DFFITS and DFBETAS — measure how much the fitted value or individual coefficient changes when a point is deleted.

You don't need to memorize these advanced formulas now, but it's useful to know they exist. The intuition you're building — that extreme x-values combined with large residuals spell trouble — is exactly the same intuition that underlies every one of these formal tools. Mastering the visual and remove-and-compare approach now will make the transition to these numerical diagnostics straightforward later.

SECTION 9

Practice Problems

PROBLEM 1 — CONCEPTUAL

Explain the difference between an outlier and an influential point. Can a point be one without being the other? Give an example situation for each case.

PROBLEM 2 — BASIC CALCULATION

A regression line is ŷ = 10 + 3x. A data point has coordinates (5, 30). Calculate the residual and state whether the point is above or below the line.

PROBLEM 3 — INTERMEDIATE

A dataset of 8 points has x̄ = 6. With all 8 points, the regression equation is ŷ = 20 + 4x (r² = 0.91). When a point at (15, 35) is removed, the equation becomes ŷ = 18 + 4.8x (r² = 0.95). Classify the point at (15, 35): is it a high-leverage point, an outlier, an influential point, or some combination? Justify your answer.

PROBLEM 4 — APPLIED

A car dealership records the age (years) and price ($) of 20 used cars. The regression model is ŷ = 28,000 − 2,100x with r² = 0.87. One car is 15 years old and priced at $22,000. The other 19 cars range from 1 to 8 years old. When the 15-year-old car is removed, the model becomes ŷ = 30,500 − 2,800x with r² = 0.93. Explain what happened and what you would recommend to the analyst.

PROBLEM 5 — CRITICAL THINKING

A student argues: 'We should always remove influential points because they make the model worse.' Do you agree or disagree? Write a short paragraph defending your position, using at least two specific reasons.

SUMMARY

Lesson Summary

An outlier is a data point with a large residual — its observed y-value is far from the value predicted by the regression line. A high-leverage point has an x-value far from the mean of x, giving it the potential to swing the line. When a point is both an outlier and a high-leverage point, it becomes an influential point — removing it causes a noticeable change in the slope, intercept, or r² of the regression model.

To identify these points, compare the regression equation with all data included to the equation after removing the suspect point. If the model changes substantially, the point is influential. Always investigate why a point is unusual — it might be a data error, a measurement from a different population, or a genuine extreme value — before deciding whether to include it in your analysis. Reporting results both with and without the influential point is the most responsible practice.