Historical Context & Motivation
Long before computers, humans needed ways to sift through information and draw conclusions. Census takers in ancient Rome counted citizens by hand, merchants in the Middle Ages tallied goods in ledger books, and 19th-century scientists organized observations into tables. The fundamental challenge has always been the same: raw data is messy, and people need transformed summaries that tell a clear story.
The core question this lesson addresses is straightforward: given a collection of rows—like a spreadsheet of student grades, weather readings, or game scores—how do you write a Python program that filters out rows you don't need, computes new fields from existing data, and prints a readable report that anyone can understand?
Core Principles & Definitions
Before you write a single line of code, it helps to understand the vocabulary and the mental model behind dataset transformation. A dataset is simply an organized collection of records—think of a spreadsheet where each row is a record and each column is a field. In Python, we commonly represent a dataset as a list of dictionaries, where each dictionary stores one row's key–value pairs. The three core operations you'll master are filtering, computing, and reporting.
Filtering Rows
if statement to keep or discard each row.Computing New Fields
Outputting a Report
print() with f-strings, alignment, and separators so the output looks organized.The Pipeline Pattern
Visual Explanation — The Transformation Pipeline
Notice how each stage in the diagram has a single, clear responsibility. The filter stage does not compute grades, and the compute stage does not worry about formatting. This separation of concerns is one of the most important habits in programming. When you keep your logic modular, you can change the filter condition—say, passing score of 70 instead of 60—without touching the grading or reporting code.
How It Works — Python Mechanics
Let's break down the Python techniques that power each stage. We'll use built-in data structures—lists and dictionaries—rather than external libraries, so you understand what's happening at every level.
Representing a Dataset
A dataset in Python is most often stored as a list of dictionaries. Each dictionary represents one row, with keys acting as column headers and values holding the cell data. For example:
students = [
{"name": "Ali", "score": 92},
{"name": "Bo", "score": 55},
{"name": "Cal", "score": 78},
{"name": "Di", "score": 41},
{"name": "Eve", "score": 88}
]
Each dictionary uses the same keys ("name" and "score"), ensuring consistency across rows.Filtering with a For Loop
To filter rows, you iterate through the list and test each dictionary against a condition. Rows that pass the test are appended to a new list. The general pattern is:
filtered = []
for row in students:
if row["score"] >= 60:
filtered.append(row)
After this loop, filtered contains only the rows where score is 60 or above.Computing New Fields
To compute a new field, you loop through the filtered list and add a new key–value pair to each dictionary. Here, we derive a letter grade:
for row in filtered:
s = row["score"]
if s >= 90:
row["grade"] = "A"
elif s >= 80:
row["grade"] = "B"
elif s >= 70:
row["grade"] = "C"
elif s >= 60:
row["grade"] = "D"
else:
row["grade"] = "F"
Now every dictionary in filtered has a third key called "grade".Outputting a Readable Report
The final stage uses f-strings (formatted string literals) and alignment specifiers to produce clean output. The :<10 syntax left-aligns text in a 10-character-wide column, while :>5 right-aligns a number in 5 characters:
print("--- Grade Report ---")
print(f"{'Name':<10} {'Score':>5} Grade")
print("-" * 26)
for row in filtered:
print(f"{row['name']:<10} {row['score']:>5} {row['grade']}")
print("-" * 26)
avg = sum(r['score'] for r in filtered) / len(filtered)
print(f"Average: {avg:.1f} | Count: {len(filtered)}")Common Transformation Patterns
Filtering and computing new fields are the two transformations you'll use most often, but there are several common variations worth knowing. The diagram below illustrates five frequently used patterns side by side, showing what each pattern does to a small sample dataset.
| Pattern | Purpose | Python Idiom | Output Shape |
|---|---|---|---|
| Filter | Keep only rows matching a condition | if row[key] >= val | Fewer rows, same fields |
| Compute | Derive new values from existing fields | row["new"] = f(row) | Same rows, more fields |
| Aggregate | Summarize many rows into one value | sum() / len() | Single value |
| Sort | Reorder rows by a field | sorted(data, key=...) | Same rows, new order |
| Format | Pretty-print for human readers | f"{val:<10}" | Text output |
Worked Example — Student Grade Report
Let's walk through a complete program from start to finish. The task: given a list of students with names and numeric scores, produce a report that shows only passing students (score ≥ 60), their letter grades, and the class average of the passing group.
"name" and "score" keys.
students = [
{"name": "Ali", "score": 92},
{"name": "Bo", "score": 55},
{"name": "Cal", "score": 78},
{"name": "Di", "score": 41},
{"name": "Eve", "score": 88}
]passing = []
for student in students:
if student["score"] >= 60:
passing.append(student)
Bo (55) and Di (41) are excluded."grade" key based on score ranges.
for student in passing:
s = student["score"]
if s >= 90:
student["grade"] = "A"
elif s >= 80:
student["grade"] = "B"
elif s >= 70:
student["grade"] = "C"
elif s >= 60:
student["grade"] = "D"
Ali gets A, Eve gets B, Cal gets C.sum() and divide by the count.
total = sum(s["score"] for s in passing)
avg = total / len(passing)
Total = 92 + 78 + 88 = 258. Average = 258 ÷ 3 = 86.0.print("=" * 30)
print(" GRADE REPORT")
print("=" * 30)
print(f"{'Name':<10} {'Score':>5} Grade")
print("-" * 30)
for s in passing:
print(f"{s['name']:<10} {s['score']:>5} {s['grade']}")
print("-" * 30)
print(f"Average: {avg:.1f} | Count: {len(passing)}")
print("=" * 30)
The output will be:
==============================
GRADE REPORT
==============================
Name Score Grade
------------------------------
Ali 92 A
Cal 78 C
Eve 88 B
------------------------------
Average: 86.0 | Count: 3
==============================Comparing Approaches — Loops vs. Comprehensions
Python gives you two major ways to filter and transform data: traditional for loops (which you've already seen) and list comprehensions (a more compact syntax). Both produce the same result, but they differ in readability, flexibility, and conciseness. Understanding both will help you choose the right tool for each situation.
| Feature | For Loop | List Comprehension |
|---|---|---|
| Syntax | result = []
for r in data:
if cond:
result.append(r) | [r for r in data if cond] |
| Lines of code | 3–4 lines | 1 line |
| Readability | Very clear for beginners; logic is explicit | Concise; can be harder to read if complex |
| Side effects | Easy to add print statements, counters, or error handling inside the loop | No room for extra logic—purely builds a list |
| Best for | Multi-step transformations, debugging, complex logic | Simple filters and maps where the logic is a single expression |
Connection to Advanced Tools & Concepts
Everything you're learning with plain Python lists and dictionaries is directly transferable to more powerful tools. When you encounter the pandas library, SQL databases, or even spreadsheet formulas, you'll recognize the same pipeline pattern: load, filter, compute, aggregate, and format. The table below maps what you know now to what you'll see next.
| Concept | This Lesson (Pure Python) | Pandas (Next Course) | SQL (Database Course) |
|---|---|---|---|
| Dataset | list of dicts | DataFrame | Table |
| Filter | if row["score"] >= 60 | df[df["score"] >= 60] | WHERE score >= 60 |
| Compute | row["grade"] = ... | df["grade"] = df.apply(...) | SELECT ..., CASE WHEN ... |
| Aggregate | sum() / len() | df["score"].mean() | AVG(score) |
| Sort | sorted(data, key=...) | df.sort_values("score") | ORDER BY score |
Beyond specific tools, the skills you're building here connect to broader computer science ideas. Abstraction—hiding complex details behind a simple interface—is what a function like get_grade(score) achieves. Decomposition—breaking a big problem into smaller, manageable steps—is exactly what the pipeline pattern does. These are the same principles that guide software engineering at every level, from a class assignment to a billion-dollar tech company.
Practice Problems
Work through these five problems, which increase in difficulty. For each one, try writing the code yourself before checking the answer. Use the dataset and pipeline patterns from the lesson.
age >= 18. After filtering, 62 rows remain. You then compute a new field called "category" for each remaining row. How many rows have the "category" field, and how many fields does each of those rows have compared to the original?products = [
{"item": "Pen", "price": 1.50},
{"item": "Notebook", "price": 12.00},
{"item": "Backpack", "price": 45.00},
{"item": "Eraser", "price": 0.75}
]
How many items will be in the filtered list?products list from Problem 2, write code that: (a) filters to keep items priced above $1.00, (b) adds a "tax" field equal to price × 0.08, and (c) adds a "total" field equal to price + tax. Show the code and the resulting data for each filtered row.weather = [
{"day": "Mon", "high": 85, "low": 62},
{"day": "Tue", "high": 91, "low": 70},
{"day": "Wed", "high": 78, "low": 58},
{"day": "Thu", "high": 95, "low": 73},
{"day": "Fri", "high": 88, "low": 65}
]
Write a complete program that: (1) computes a "avg_temp" field as (high + low) / 2, (2) filters to keep only days where avg_temp ≥ 75, (3) sorts the result by avg_temp descending, and (4) prints a formatted report with day, high, low, and avg_temp columns.data = [
{"name": "X", "val": 10},
{"name": "Y", "val": 20},
{"name": "Z", "val": 30}
]
filtered = []
for row in data:
if row["val"] > 15:
row["double"] = row["val"] * 2
filtered.append(row)
print(data[1])
The student expects data[1] to still have only two keys ("name" and "val"), but it actually prints three keys including "double". Explain why this happens and propose a fix that keeps the original data list unchanged.Lesson Summary
In this lesson you learned to transform datasets using a three-stage pipeline pattern in Python. You represent data as a list of dictionaries, where each dictionary is a row and each key is a field. The filter step uses a for loop with an if condition to select rows, producing a shorter list with the same fields. The compute step adds new key–value pairs to each dictionary—deriving values like letter grades or tax amounts from existing data. The report step uses f-strings with alignment specifiers to print clean, human-readable output.
You also explored five common transformation patterns—filter, map/compute, aggregate, sort, and format—and compared for loops with list comprehensions as two approaches to the same task. These pure-Python skills translate directly to professional tools like pandas and SQL, and they embody fundamental CS principles of abstraction and decomposition.