Data Analysis Limitations — I can explain limitations of a small dataset analysis (missing data, bias) at my level. - Computer Programming 1 Python

SECTION 1

Historical Context & Motivation

Long before Python existed, people struggled with a fundamental question: how much data do you need before you can trust your conclusions? From early census counts that excluded entire populations to modern-day recommendation algorithms trained on incomplete user data, the limitations of small datasets, missing data, and bias have shaped the history of statistics, computing, and data science. Understanding these limitations is essential for any programmer who writes code to analyze real-world information.

1854 • John Snow's Cholera Map

Dr. John Snow mapped cholera cases in London using a small, localized dataset. His analysis was groundbreaking but limited to one neighborhood, illustrating how small sample size can both reveal and conceal patterns.

1936 • The Literary Digest Poll Disaster

A magazine polled 2.4 million people about the U.S. presidential election and predicted the wrong winner. The sample was huge but biased — it over-represented wealthy Americans who owned phones and cars.

1970s • Database Systems and NULL Values

As relational databases emerged, computer scientists introduced the concept of NULL to represent missing or unknown data. This formalized the idea that datasets are often incomplete.

2010s • AI Bias Scandals

Machine learning models trained on biased datasets were found to discriminate by race and gender in hiring, lending, and criminal justice. These cases proved that biased training data produces biased results, no matter how sophisticated the algorithm.

These historical examples all point to a central challenge: when you write a Python program that analyzes data, your code can only be as reliable as the data it processes. If the dataset is too small, has gaps, or reflects a skewed perspective, your results will be limited — or even wrong. This lesson explores exactly how and why that happens.

SECTION 2

Core Principles & Definitions

Before diving into code, you need to understand three foundational ideas that explain why data analysis can go wrong. These concepts apply whether you are working with a spreadsheet of ten rows or a database of ten million records, but they become especially dangerous with smaller datasets.

Small Sample Size

A small sample is a dataset that contains too few observations to accurately represent the larger group (the population). With fewer data points, random flukes have an outsized effect on averages, percentages, and other summary statistics.

Missing Data

Missing data occurs when some values in a dataset are absent — a survey respondent skips a question, a sensor fails to record, or a spreadsheet cell is left blank. In Python, these often appear as None or NaN. Missing data can silently distort calculations.

Bias

Bias means the data systematically favors one outcome over another. Selection bias happens when certain groups are over- or under-represented. Measurement bias happens when the way data is collected consistently skews the results.

Outliers

An outlier is a data point that is far outside the typical range. In a large dataset, outliers are diluted. In a small dataset, a single outlier can dramatically shift the mean or trend, giving a misleading picture of what is 'normal.'

✦

KEY TAKEAWAY

Think of data analysis like tasting soup. If you stir the pot well and take a big spoonful, you get a good sense of the flavor. But if you only dip a tiny spoon into one corner of the pot, you might taste only broth — or only a chunk of potato. A small, biased, or incomplete sample is like that tiny, unstirred spoonful: it does not represent the whole.

SECTION 3

Visual Explanation — How Small Samples Mislead

The diagram below compares two scenarios: analyzing a small dataset of five students versus a larger dataset of thirty students. Notice how the same question — "What is the average test score?" — can produce very different answers depending on sample size and whether any data is missing.

Comparison of a five-student sample versus a thirty-student sample. The small sample has one outlier (score of 45) that drags the mean down by roughly 7 points, and one missing value that represents 20% of the data. In the larger sample, the same outlier barely moves the needle, and two missing values are only about 7% of the total.

The key visual takeaway is that the small sample's average (78.75) is substantially lower than the larger sample's average (≈85.5), primarily because one outlier has a disproportionate effect. Furthermore, the missing value in the small sample represents a full 20% of the data — removing it changes the story. In the large sample, two missing values represent less than 7%, so the overall picture remains stable. When you write Python code like sum(scores) / len(scores), the computer does not know whether your data is trustworthy. It simply calculates. That is why understanding these limitations is your job as the programmer.

SECTION 4

How Small Datasets Distort Analysis in Python

When you run data analysis in Python, your code follows precise arithmetic rules. The problem is that those rules assume the data is complete and representative. Let's examine the math behind why small datasets are risky, using formulas you would actually compute in a Python program.

SAMPLE MEAN

x̄ = (x₁ + x₂ + ... + xₙ) ÷ n

Where x̄ is the sample mean, each xᵢ is an individual data point, and n is the number of data points. When n is small, each individual value has a large influence on the mean. If n = 5, each value controls 20% of the result.

IMPACT OF ONE OUTLIER

Shift in mean = (outlier − x̄_without) ÷ n

This shows how much a single outlier moves the average. With n = 5 and an outlier 40 points below the group average, the mean shifts by 40 ÷ 5 = 8 points. With n = 30, the same outlier shifts the mean by only 40 ÷ 30 ≈ 1.3 points.

MISSING DATA PERCENTAGE

Missing % = (number of missing values ÷ total expected values) × 100

If 1 out of 5 values is missing, you have 20% missing data. If 1 out of 100 is missing, you have 1%. Analysts often consider more than 5−10% missing data as a warning sign that results may be unreliable.

⚠

Python Pitfall

In Python, if you have a list like scores = [92, 88, None, 45, 90] and you try sum(scores), you will get a TypeError. You must filter out None values first using something like clean = [s for s in scores if s is not None]. But filtering silently reduces your sample size, which amplifies all the problems we've discussed.

SECTION 5

Types of Bias and Missing Data in Practice

Not all data problems look the same. The diagram below maps out the most common types of limitations you will encounter when analyzing datasets in Python projects. Understanding each type helps you recognize problems in your own code and communicate them clearly when presenting results.

A taxonomy showing three major categories of data limitations. Small sample size leads to outlier sensitivity and low confidence. Missing data can be random (MCAR) or systematic (MNAR). Bias includes selection bias and measurement bias. All three can overlap, compounding their effects.

Pay special attention to the difference between MCAR (Missing Completely At Random) and MNAR (Missing Not At Random). MCAR is like a student accidentally skipping a survey question — the missingness has nothing to do with the answer. MNAR is more dangerous: imagine students who scored poorly on a test refusing to report their scores. The missing data is related to the value itself, which means ignoring it will make your average higher than reality. In Python, both look the same (None in a list), but their effects on your analysis are completely different.

Limitation	Example in a School Project	Effect on Python Output
Small sample	Surveying 5 classmates about lunch preferences instead of the full school	Percentages swing wildly; one person's choice = 20%
Missing data (MCAR)	A few students forgot to fill in their grade level on a survey	Slightly reduces sample; minor impact if few values missing
Missing data (MNAR)	Students who failed a test don't submit their scores	Average appears higher than it really is; systematic error
Selection bias	Posting a survey only in the AP class group chat	Results reflect high-achieving students, not the whole school
Outlier in small set	One student reports spending 10 hours on homework when others spend 1−2	Mean homework time is inflated; misleading conclusion

Common limitations with school-based examples

SECTION 6

Worked Example — Analyzing a Small Survey in Python

Imagine you surveyed 8 students about how many hours per week they spend on social media. Some students didn't answer, and you want to calculate the average. Let's walk through the analysis step by step, identifying every limitation along the way.

Social Media Hours — Small Survey Analysis

1
Step 1 — Collect the Raw Data

You store the survey responses in a Python list. Two students did not respond, so you use None for their entries.

hours = [3, 5, None, 2, 14, 4, None, 3]
2
Step 2 — Identify Limitations Before Computing

Before calculating anything, you should note the problems. The sample is only 8 students out of a school of hundreds (small sample size). Two values are missing (25% missing data). The value 14 looks like a potential outlier — it is much higher than the others. You also asked only students in your friend group, which may introduce selection bias.
3
Step 3 — Clean the Data in Python

Filter out None values using a list comprehension. This leaves you with only 6 data points.

clean = [h for h in hours if h is not None] # [3, 5, 2, 14, 4, 3]
4
Step 4 — Calculate the Mean

Compute the average: (3 + 5 + 2 + 14 + 4 + 3) ÷ 6 = 31 ÷ 6.

Mean = 5.17 hours
5
Step 5 — Assess the Outlier's Impact

Remove the outlier (14) and recalculate: (3 + 5 + 2 + 4 + 3) ÷ 5 = 17 ÷ 5 = 3.4. The outlier inflated the mean by nearly 2 hours (5.17 − 3.4 = 1.77). In your report, you should mention that one outlier shifted the average by 52%.

Mean without outlier = 3.4 hours
6
Step 6 — Write a Limitations Statement

Any responsible data analysis should include a limitations section. Here is an example: "This analysis is based on 6 usable responses out of 8 surveyed (25% missing data). The sample represents less than 1% of the student body and was drawn from a single friend group, introducing potential selection bias. One outlier value (14 hours) significantly affects the mean."

Always state your limitations — it makes your analysis more trustworthy, not less.

SECTION 7

Strategies for Handling Limitations

Knowing the limitations is only half the battle. The other half is knowing what to do about them. The table below compares common strategies for addressing each type of limitation, along with their tradeoffs.

Strategy	What It Helps With	Tradeoff / Risk
Collect more data	Small sample size, low confidence	Takes more time and resources; not always possible
Use median instead of mean	Outlier sensitivity	Median ignores magnitude of extreme values, which might actually matter
Fill missing values (imputation)	Missing data reducing sample size	Invented data may introduce new inaccuracies; works better for MCAR than MNAR
Randomize data collection	Selection bias	Truly random sampling is hard to achieve in practice
Report limitations transparently	All types — does not fix the problem but prevents misinterpretation	No tradeoff — this should always be done

Strategies for mitigating data limitations

✦

KEY TAKEAWAY

Think of these strategies like safety gear. Wearing a helmet doesn't guarantee you won't get hurt in a bike crash, but it drastically reduces the damage. Similarly, using the median, collecting more data, and being transparent about limitations won't make imperfect data perfect, but they make your analysis much more responsible and useful.

SECTION 8

Connection to Professional Data Science

The concepts you are learning now — recognizing small samples, missing data, and bias — are the exact same issues that professional data scientists deal with every day, just at a larger scale and with more sophisticated tools. The table below shows how the ideas from this lesson connect to advanced techniques you might encounter in college or a career.

What You Learn Now	Advanced Version
Recognizing small sample size	Statistical power analysis — calculating the minimum sample size needed for reliable results
Filtering out `None` values	Multiple imputation, expectation-maximization algorithms for estimating missing values
Noticing selection bias in friend-group surveys	Stratified random sampling, weighting survey responses, fairness audits in machine learning
Using median to reduce outlier impact	Robust statistics, trimmed means, winsorization, anomaly detection algorithms
Writing a limitations statement	Publishing confidence intervals, p-values, and effect sizes in research papers

From high school concepts to professional data science

Libraries like pandas and scikit-learn in Python provide built-in tools for handling missing data (like df.dropna() and df.fillna()) and detecting outliers. But even these professional tools cannot fix fundamentally biased or insufficient data. The critical thinking skills you develop now — asking "Is this dataset good enough to answer my question?" — are what separate a competent programmer from someone who just runs code.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

A student surveys 4 classmates about their favorite programming language and concludes that "75% of high school students prefer Python." Identify at least two data analysis limitations in this claim.

Show answer

First, the sample size is extremely small (n = 4), meaning one person's answer controls 25% of the result. This makes the percentage highly unstable — surveying a different group of 4 students could easily produce "75% prefer JavaScript" instead. Second, the student surveyed only their own classmates, which introduces selection bias. Classmates in the same computer science class are more likely to have been exposed to Python, so they do not represent all high school students. A valid conclusion would need a much larger, randomly selected sample from multiple schools and classes.

Question 2: BASIC CALCULATION

Given the Python list temps = [72, 68, None, 75, 71, None, 73], calculate the mean after removing None values. Then state what percentage of the data is missing and whether this is a concern.

Show answer

After filtering out the two None values, the clean list is [72, 68, 75, 71, 73]. The mean is (72 + 68 + 75 + 71 + 73) ÷ 5 = 359 ÷ 5 = 71.8. The missing data percentage is 2 ÷ 7 × 100 ≈ 28.6%. This is a significant concern because nearly 29% of the data is absent. If the missing temperatures were unusually high or low, the true mean could be quite different from 71.8. With such a small sample (only 7 total, 5 usable), each value has a large impact on the result.

Question 3: INTERMEDIATE

You collect test scores from 6 students: [88, 92, 85, 90, 15, 91]. Calculate both the mean and the median. Explain which measure better represents the typical student's performance and why.

Show answer

The mean is (88 + 92 + 85 + 90 + 15 + 91) ÷ 6 = 461 ÷ 6 ≈ 76.8. To find the median, sort the list: [15, 85, 88, 90, 91, 92]. With 6 values, the median is the average of the 3rd and 4th values: (88 + 90) ÷ 2 = 89.0. The median of 89.0 is a much better representation of the typical student's performance because the outlier score of 15 dramatically pulls the mean down to 76.8, which is lower than five of the six students' actual scores. In a small dataset like this, the median is more resistant to outliers and gives a more honest picture of central tendency.

Question 4: APPLIED

You are building a Python program to recommend the best restaurant in your town based on online reviews. Your dataset has 200 reviews, but 180 of them are from people under age 25 because the reviews were collected from a social media app. You notice that the top-rated restaurant is a trendy bubble tea shop. Write a short limitations paragraph for your project report, identifying at least two specific problems and suggesting one improvement.

Show answer

Example limitations paragraph: "This analysis is subject to significant selection bias because 90% of the reviews (180 out of 200) come from users under age 25, likely due to the demographic makeup of the social media platform used for data collection. This means the preferences of older adults, families, and seniors are severely underrepresented. As a result, the recommendation of the bubble tea shop may reflect the tastes of a narrow age group rather than the broader community. Additionally, the dataset may suffer from self-selection bias, as people who take time to write reviews tend to have strong opinions (either very positive or very negative), leaving moderate opinions unrepresented. One improvement would be to supplement the social media data with reviews from other platforms that attract a wider age range, such as Google Reviews or Yelp."

Question 5: CRITICAL THINKING

Consider a dataset where students' reported study hours are: [2, 3, None, 2, None, None, 3, 1]. The three None values all belong to students who admitted they "didn't want to say" how little they studied. Is this missing data MCAR or MNAR? Explain how this affects the mean, and propose two different approaches your Python program could take — describing the tradeoff of each.

Show answer

This is MNAR (Missing Not At Random) because the reason the data is missing is directly related to the value itself — students who studied very little were embarrassed to report it. This means the missing values are likely lower than the observed values. The current mean of the available data is (2 + 3 + 2 + 3 + 1) ÷ 5 = 2.2 hours, but the true mean is probably lower. Approach 1: Simply drop the missing values and report the mean as 2.2, but include a clear disclaimer that the result is likely inflated because low-study students self-censored. The tradeoff is honesty — you acknowledge the bias but cannot fix it numerically. Approach 2: Impute the missing values with a low estimate (e.g., 0 or 1 hour, based on the assumption that these students studied very little). This gives a mean of (2 + 3 + 0 + 2 + 0 + 0 + 3 + 1) ÷ 8 = 1.375 or (2 + 3 + 1 + 2 + 1 + 1 + 3 + 1) ÷ 8 = 1.75. The tradeoff is that you are guessing — your imputed values might be wrong, and you have introduced assumptions into supposedly objective data. Neither approach is perfect, which is exactly the point: MNAR data cannot be fixed purely with code.

SUMMARY

Lesson Summary

Summary

Data analysis in Python is only as reliable as the data you feed it. A small sample size makes your results vulnerable to random fluctuations and outliers — a single extreme value can drag your mean far from the truth. Missing data silently shrinks your usable sample and, when the missingness is systematic (MNAR), actively distorts your results. Bias — whether selection bias or measurement bias — means your data does not fairly represent the population you are trying to study.

To work responsibly with data in Python, you should clean your data by filtering None values, consider using the median instead of the mean when outliers are present, and strive to collect larger, more representative samples whenever possible. Most importantly, always report your limitations honestly. Stating what your analysis cannot tell you is just as important as stating what it can — and it is the hallmark of a thoughtful, ethical programmer.