Transforming Datasets — I can transform a dataset (filter rows, compute new fields) and output a readable report. - Computer Programming 1 Python

SECTION 1

Historical Context & Motivation

Long before computers, humans needed ways to sift through information and draw conclusions. Census takers in ancient Rome counted citizens by hand, merchants in the Middle Ages tallied goods in ledger books, and 19th-century scientists organized observations into tables. The fundamental challenge has always been the same: raw data is messy, and people need transformed summaries that tell a clear story.

1890 • Hollerith Punch Cards

Herman Hollerith invents electromechanical tabulating machines to process U.S. Census data, reducing years of hand-counting to months. This marks the birth of automated data processing.

1970 • Relational Databases & SQL

Edgar Codd proposes the relational model, leading to SQL—a language designed to filter rows and compute new columns from structured tables.

1991 • Python Released

Guido van Rossum releases Python, a language prized for readability. Its built-in lists and dictionaries make it ideal for quick data manipulation scripts.

2008 • Pandas Library

Wes McKinney creates the pandas library, bringing powerful data-frame operations to Python. However, pure Python lists and dictionaries remain the foundation for learning how transformations work under the hood.

Today • Data Everywhere

From social media analytics to climate science, transforming datasets is a daily task for millions of programmers. The skills you learn here apply whether you use raw Python or advanced libraries.

The core question this lesson addresses is straightforward: given a collection of rows—like a spreadsheet of student grades, weather readings, or game scores—how do you write a Python program that filters out rows you don't need, computes new fields from existing data, and prints a readable report that anyone can understand?

SECTION 2

Core Principles & Definitions

Before you write a single line of code, it helps to understand the vocabulary and the mental model behind dataset transformation. A dataset is simply an organized collection of records—think of a spreadsheet where each row is a record and each column is a field. In Python, we commonly represent a dataset as a list of dictionaries, where each dictionary stores one row's key–value pairs. The three core operations you'll master are filtering, computing, and reporting.

Filtering Rows

Selecting only the records that meet a condition. In Python, this means iterating through a list and using an if statement to keep or discard each row.

Computing New Fields

Deriving new information from existing columns—for example, calculating a letter grade from a numeric score, or computing profit from revenue minus cost.

Outputting a Report

Printing results in a formatted, human-readable way using print() with f-strings, alignment, and separators so the output looks organized.

The Pipeline Pattern

Most real-world scripts follow a three-step pipeline: load data → transform data → output results. Keeping these stages separate makes your code easier to read, test, and reuse.

✦

KEY TAKEAWAY

Think of dataset transformation like cooking. Your raw dataset is the pile of ingredients on the counter. Filtering is picking out only the ingredients you need. Computing new fields is chopping, mixing, and seasoning—creating something new from what you have. Outputting a report is plating the dish so it looks appetizing. The pipeline pattern is your recipe card that keeps the whole process organized.

SECTION 3

Visual Explanation — The Transformation Pipeline

The pipeline flows left-to-right, then downward. The raw dataset (blue box) holds five student records. The filter step (violet box) removes students below 60. The compute step (pink box) adds a letter grade. Finally, the report (cyan box) formats everything neatly for a human reader.

Notice how each stage in the diagram has a single, clear responsibility. The filter stage does not compute grades, and the compute stage does not worry about formatting. This separation of concerns is one of the most important habits in programming. When you keep your logic modular, you can change the filter condition—say, passing score of 70 instead of 60—without touching the grading or reporting code.

SECTION 4

How It Works — Python Mechanics

Let's break down the Python techniques that power each stage. We'll use built-in data structures—lists and dictionaries—rather than external libraries, so you understand what's happening at every level.

Representing a Dataset

A dataset in Python is most often stored as a list of dictionaries. Each dictionary represents one row, with keys acting as column headers and values holding the cell data. For example:

ℹ

Data Structure Pattern

students = [
  {"name": "Ali",  "score": 92},
  {"name": "Bo",   "score": 55},
  {"name": "Cal",  "score": 78},
  {"name": "Di",   "score": 41},
  {"name": "Eve",  "score": 88}
]

Each dictionary uses the same keys ("name" and "score"), ensuring consistency across rows.

Filtering with a For Loop

To filter rows, you iterate through the list and test each dictionary against a condition. Rows that pass the test are appended to a new list. The general pattern is:

💡

Filter Pattern

filtered = []
for row in students:
    if row["score"] >= 60:
        filtered.append(row)

After this loop, filtered contains only the rows where score is 60 or above.

Computing New Fields

To compute a new field, you loop through the filtered list and add a new key–value pair to each dictionary. Here, we derive a letter grade:

💡

Compute Pattern

for row in filtered:
    s = row["score"]
    if s >= 90:
        row["grade"] = "A"
    elif s >= 80:
        row["grade"] = "B"
    elif s >= 70:
        row["grade"] = "C"
    elif s >= 60:
        row["grade"] = "D"
    else:
        row["grade"] = "F"

Now every dictionary in filtered has a third key called "grade".

Outputting a Readable Report

The final stage uses f-strings (formatted string literals) and alignment specifiers to produce clean output. The :<10 syntax left-aligns text in a 10-character-wide column, while :>5 right-aligns a number in 5 characters:

💡

Report Pattern

print("--- Grade Report ---")
print(f"{'Name':<10} {'Score':>5}  Grade")
print("-" * 26)
for row in filtered:
    print(f"{row['name']:<10} {row['score']:>5}  {row['grade']}")
print("-" * 26)
avg = sum(r['score'] for r in filtered) / len(filtered)
print(f"Average: {avg:.1f}  |  Count: {len(filtered)}")

SECTION 5

Common Transformation Patterns

Filtering and computing new fields are the two transformations you'll use most often, but there are several common variations worth knowing. The diagram below illustrates five frequently used patterns side by side, showing what each pattern does to a small sample dataset.

Five transformation patterns are shown as colored cards. Filter selects rows. Map/Compute adds fields. Aggregate reduces many rows to a single summary value. Sort reorders rows. Format produces polished output. The reference box below the cards shows the one-line Python syntax for each.

Pattern	Purpose	Python Idiom	Output Shape
Filter	Keep only rows matching a condition	`if row[key] >= val`	Fewer rows, same fields
Compute	Derive new values from existing fields	`row["new"] = f(row)`	Same rows, more fields
Aggregate	Summarize many rows into one value	`sum() / len()`	Single value
Sort	Reorder rows by a field	`sorted(data, key=...)`	Same rows, new order
Format	Pretty-print for human readers	`f"{val:<10}"`	Text output

Summary of five common transformation patterns and their effects on data shape.

SECTION 6

Worked Example — Student Grade Report

Let's walk through a complete program from start to finish. The task: given a list of students with names and numeric scores, produce a report that shows only passing students (score ≥ 60), their letter grades, and the class average of the passing group.

Full Pipeline — Grade Report Generator

1
Step 1 — Define the Raw Dataset

Create a list of dictionaries, where each dictionary has "name" and "score" keys. students = [ {"name": "Ali", "score": 92}, {"name": "Bo", "score": 55}, {"name": "Cal", "score": 78}, {"name": "Di", "score": 41}, {"name": "Eve", "score": 88} ]

5 rows, each with 2 fields
2
Step 2 — Filter: Keep Passing Students

Loop through the list and keep only students with a score of 60 or above. We build a new list so the original data stays intact. passing = [] for student in students: if student["score"] >= 60: passing.append(student) Bo (55) and Di (41) are excluded.

passing contains 3 rows: Ali, Cal, Eve
3
Step 3 — Compute: Add Letter Grades

Iterate through the filtered list and assign a "grade" key based on score ranges. for student in passing: s = student["score"] if s >= 90: student["grade"] = "A" elif s >= 80: student["grade"] = "B" elif s >= 70: student["grade"] = "C" elif s >= 60: student["grade"] = "D" Ali gets A, Eve gets B, Cal gets C.

Each dict now has 3 keys: name, score, grade
4
Step 4 — Aggregate: Compute the Average

Use a generator expression inside sum() and divide by the count. total = sum(s["score"] for s in passing) avg = total / len(passing) Total = 92 + 78 + 88 = 258. Average = 258 ÷ 3 = 86.0.

avg = 86.0
5
Step 5 — Output: Print the Report

Use f-strings with alignment specifiers to create neat columns. print("=" * 30) print(" GRADE REPORT") print("=" * 30) print(f"{'Name':<10} {'Score':>5} Grade") print("-" * 30) for s in passing: print(f"{s['name']:<10} {s['score']:>5} {s['grade']}") print("-" * 30) print(f"Average: {avg:.1f} | Count: {len(passing)}") print("=" * 30) The output will be: ============================== GRADE REPORT ============================== Name Score Grade ------------------------------ Ali 92 A Cal 78 C Eve 88 B ------------------------------ Average: 86.0 | Count: 3 ==============================

Readable, aligned report printed to the console

💡

WHY THIS MATTERS

Each step produces a clear intermediate result that you can inspect and debug. If the grades look wrong, you check Step 3. If the wrong students appear, you check Step 2. This pipeline approach is how professional developers structure data scripts—whether in Python, SQL, or spreadsheets.

SECTION 7

Comparing Approaches — Loops vs. Comprehensions

Python gives you two major ways to filter and transform data: traditional for loops (which you've already seen) and list comprehensions (a more compact syntax). Both produce the same result, but they differ in readability, flexibility, and conciseness. Understanding both will help you choose the right tool for each situation.

Feature	For Loop	List Comprehension
Syntax	`result = [] for r in data: if cond: result.append(r)`	`[r for r in data if cond]`
Lines of code	3–4 lines	1 line
Readability	Very clear for beginners; logic is explicit	Concise; can be harder to read if complex
Side effects	Easy to add print statements, counters, or error handling inside the loop	No room for extra logic—purely builds a list
Best for	Multi-step transformations, debugging, complex logic	Simple filters and maps where the logic is a single expression

Comparison of for-loop and list-comprehension approaches for dataset transformations.

✦

KEY TAKEAWAY

Think of a for loop like writing step-by-step driving directions, and a list comprehension like giving a single GPS coordinate. Both get you there, but the step-by-step version is easier to follow when the route is complicated. Start with for loops until you're comfortable, then adopt comprehensions for simple one-liners.

💡

Pro Tip: Combine Them

You can use a comprehension for the filter step and a for loop for the compute step. There's no rule that says your entire pipeline must use one style. Mixing approaches is perfectly normal in real-world Python code, and professional developers often do exactly this.

SECTION 8

Connection to Advanced Tools & Concepts

Everything you're learning with plain Python lists and dictionaries is directly transferable to more powerful tools. When you encounter the pandas library, SQL databases, or even spreadsheet formulas, you'll recognize the same pipeline pattern: load, filter, compute, aggregate, and format. The table below maps what you know now to what you'll see next.

Concept	This Lesson (Pure Python)	Pandas (Next Course)	SQL (Database Course)
Dataset	`list of dicts`	`DataFrame`	Table
Filter	`if row["score"] >= 60`	`df[df["score"] >= 60]`	`WHERE score >= 60`
Compute	`row["grade"] = ...`	`df["grade"] = df.apply(...)`	`SELECT ..., CASE WHEN ...`
Aggregate	`sum() / len()`	`df["score"].mean()`	`AVG(score)`
Sort	`sorted(data, key=...)`	`df.sort_values("score")`	`ORDER BY score`

How the same transformation concepts appear in Python, pandas, and SQL.

Beyond specific tools, the skills you're building here connect to broader computer science ideas. Abstraction—hiding complex details behind a simple interface—is what a function like get_grade(score) achieves. Decomposition—breaking a big problem into smaller, manageable steps—is exactly what the pipeline pattern does. These are the same principles that guide software engineering at every level, from a class assignment to a billion-dollar tech company.

SECTION 9

Practice Problems

Work through these five problems, which increase in difficulty. For each one, try writing the code yourself before checking the answer. Use the dataset and pipeline patterns from the lesson.

Question 1: CONCEPTUAL

A dataset has 100 rows. You apply a filter that keeps only rows where age >= 18. After filtering, 62 rows remain. You then compute a new field called "category" for each remaining row. How many rows have the "category" field, and how many fields does each of those rows have compared to the original?

Show answer

Exactly 62 rows will have the "category" field, because the compute step operates on the filtered list. Each of those 62 rows now has one more field than the originals—if the original rows had, say, 4 fields, they now have 5. The 38 rows that were filtered out are untouched and have no "category" key. Filtering reduces the number of rows; computing increases the number of fields per row.

Question 2: BASIC CALCULATION

Given this dataset, write a for loop that creates a new list containing only the products with a price above 10.00:

products = [
    {"item": "Pen", "price": 1.50},
    {"item": "Notebook", "price": 12.00},
    {"item": "Backpack", "price": 45.00},
    {"item": "Eraser", "price": 0.75}
]

How many items will be in the filtered list?

Show answer

The filter code is:

expensive = []
for p in products:
    if p["price"] > 10.00:
        expensive.append(p)

Only Notebook (12.00) and Backpack (45.00) have prices above 10.00, so expensive contains 2 items. The Pen (1.50) and Eraser (0.75) are excluded because their prices do not satisfy the condition price > 10.00.

Question 3: INTERMEDIATE

Using the products list from Problem 2, write code that: (a) filters to keep items priced above $1.00, (b) adds a "tax" field equal to price × 0.08, and (c) adds a "total" field equal to price + tax. Show the code and the resulting data for each filtered row.

Show answer

filtered = []
for p in products:
    if p["price"] > 1.00:
        filtered.append(p)

for p in filtered:
    p["tax"] = round(p["price"] * 0.08, 2)
    p["total"] = round(p["price"] + p["tax"], 2)

Three items pass the filter: Pen (1.50) is included because 1.50 > 1.00. Results: Pen → tax 0.12, total 1.62; Notebook → tax 0.96, total 12.96; Backpack → tax 3.60, total 48.60. Using round() prevents floating-point rounding issues like 0.120000000001.

Question 4: APPLIED

You are given weather data for a week:

weather = [
    {"day": "Mon", "high": 85, "low": 62},
    {"day": "Tue", "high": 91, "low": 70},
    {"day": "Wed", "high": 78, "low": 58},
    {"day": "Thu", "high": 95, "low": 73},
    {"day": "Fri", "high": 88, "low": 65}
]

Write a complete program that: (1) computes a "avg_temp" field as (high + low) / 2, (2) filters to keep only days where avg_temp ≥ 75, (3) sorts the result by avg_temp descending, and (4) prints a formatted report with day, high, low, and avg_temp columns.

Show answer

# Step 1: Compute
for day in weather:
    day["avg_temp"] = (day["high"] + day["low"]) / 2

# Step 2: Filter
hot_days = [d for d in weather if d["avg_temp"] >= 75]

# Step 3: Sort
hot_days.sort(key=lambda d: d["avg_temp"], reverse=True)

# Step 4: Report
print(f"{'Day':<5} {'High':>4} {'Low':>4} {'Avg':>5}")
print("-" * 20)
for d in hot_days:
    print(f"{d['day']:<5} {d['high']:>4} {d['low']:>4} {d['avg_temp']:>5.1f}")

Avg temps: Mon 73.5, Tue 80.5, Wed 68.0, Thu 84.0, Fri 76.5. After filtering (≥ 75), Wed and Mon are removed. Sorted descending: Thu (84.0), Tue (80.5), Fri (76.5). Notice that compute happens before filter here—sometimes you need the computed field to make the filtering decision.

Question 5: CRITICAL THINKING

A student writes this code to filter a dataset and then compute a new field:

data = [
    {"name": "X", "val": 10},
    {"name": "Y", "val": 20},
    {"name": "Z", "val": 30}
]

filtered = []
for row in data:
    if row["val"] > 15:
        row["double"] = row["val"] * 2
        filtered.append(row)

print(data[1])

The student expects data[1] to still have only two keys ("name" and "val"), but it actually prints three keys including "double". Explain why this happens and propose a fix that keeps the original data list unchanged.

Show answer

This happens because Python dictionaries are mutable objects, and the list stores references to those objects—not copies of them. When the code does row["double"] = row["val"] * 2, it modifies the original dictionary that data[1] points to. Both data[1] and filtered[0] reference the same object in memory. To fix this, create a new dictionary instead of modifying the original:

filtered = []
for row in data:
    if row["val"] > 15:
        new_row = dict(row)  # shallow copy
        new_row["double"] = row["val"] * 2
        filtered.append(new_row)

Now data[1] remains untouched. This concept—mutation vs. copying—is one of the most common sources of bugs in Python.

SUMMARY

Lesson Summary

Summary

In this lesson you learned to transform datasets using a three-stage pipeline pattern in Python. You represent data as a list of dictionaries, where each dictionary is a row and each key is a field. The filter step uses a for loop with an if condition to select rows, producing a shorter list with the same fields. The compute step adds new key–value pairs to each dictionary—deriving values like letter grades or tax amounts from existing data. The report step uses f-strings with alignment specifiers to print clean, human-readable output.

You also explored five common transformation patterns—filter, map/compute, aggregate, sort, and format—and compared for loops with list comprehensions as two approaches to the same task. These pure-Python skills translate directly to professional tools like pandas and SQL, and they embody fundamental CS principles of abstraction and decomposition.