Text Statistics — I can compute simple text statistics (word/character counts, frequency) using loops/dictionaries at my level. - Computer Programming 1 Python

SECTION 1

Historical Context & Motivation

Long before computers existed, people needed to analyze text. Monks counted words in religious manuscripts to detect copying errors, and cryptographers during wartime studied how often each letter appeared in coded messages to crack enemy ciphers. The idea of text statistics — measuring quantitative properties of written language — is centuries old. What changed is that computers made these calculations instant and scalable.

Today, text statistics power everything from spam filters in your email inbox to search-engine ranking algorithms. When you use a word processor and it displays a word count at the bottom of the screen, that is text statistics at work. Understanding how to compute these measurements in code gives you a foundation for fields like natural language processing, data science, and cybersecurity.

1400s • Manuscript Word Counting

Scribes counted words in hand-copied texts to verify accuracy and detect omissions — the earliest manual form of text statistics.

1943 • Colossus Code-Breaking

British engineers built the Colossus computer to perform letter-frequency analysis on encrypted German messages during World War II.

1960s • Computational Linguistics Emerges

Researchers began using mainframe computers to analyze large text corpora, counting word frequencies to study language patterns.

1991 • Python Is Created

Guido van Rossum released Python, whose built-in string methods and dictionary data structure made text analysis accessible to beginners.

2020s • AI & NLP Everywhere

Modern AI systems like chatbots and search engines rely on text statistics as foundational building blocks for understanding human language.

The central question this lesson addresses is straightforward: given a block of text, how can you write Python code that counts its characters, splits it into words, and tallies how frequently each letter or word appears? Mastering this skill bridges the gap between knowing basic Python syntax and being able to do something genuinely useful with it.

SECTION 2

Core Principles & Definitions

Before writing any code, you need a clear vocabulary for the measurements you will compute. Each of these ideas maps directly to a small pattern in Python that you will see repeated throughout this lesson.

Character Count

The total number of individual characters in a string, including letters, digits, spaces, and punctuation. In Python, len(text) returns this value instantly.

Word Count

The number of words in a string, where a "word" is typically any sequence of characters separated by whitespace. Use text.split() to get a list of words, then len() to count them.

Frequency

How many times a specific character or word appears. A frequency dictionary maps each unique item to its count — e.g., {"a": 5, "b": 2}.

Iteration with Loops

A for loop visits each character (or each word) one at a time, allowing you to inspect, count, or categorize every element in a string.

Dictionary Accumulation

The pattern of building a dictionary inside a loop: check if a key exists, then either initialize it to 1 or increment the existing count by 1. This is the core technique for computing frequencies.

✦

KEY TAKEAWAY

Think of a frequency dictionary like a tally chart in a science class. Every time you observe a data point (a letter or word), you find its row on the chart and add one mark. In Python, the dictionary is your chart — each key is a row label, and each value is the running tally.

SECTION 3

Visual Explanation

The diagram below traces how Python processes a short string character by character, building a frequency dictionary as it goes. Follow the arrow from left to right to see each step of the loop.

Each colored box represents one pass through the for loop. Notice that on iteration 4, the key "l" already exists, so its value is incremented from 1 to 2 rather than being re-created.

The key decision point in the loop is the if/else check: is this character already a key in the dictionary? If not, create the key with a value of 1. If it is already there, add 1 to the existing value. This two-branch pattern is sometimes called the accumulator pattern, and you will use it constantly in Python programming.

SECTION 4

How It Works — The Code Patterns

There are three fundamental measurements you will compute in this lesson. Each one maps to a short, reusable Python pattern. Let's look at the logic behind each measurement, expressed as concise formulas before we write full code.

CHARACTER COUNT

char_count = len(text)

The built-in len() function returns the total number of characters in a string, including spaces and punctuation. For example, len("Hi there") returns 8.

WORD COUNT

word_count = len(text.split())

text.split() breaks the string at every whitespace boundary and returns a list of words. Wrapping that list in len() gives the number of words. For "Hi there" the result is 2.

FREQUENCY ACCUMULATION

freq[char] = freq.get(char, 0) + 1

Inside a loop, freq.get(char, 0) returns the current count for char if it exists, or 0 if it does not. Adding 1 and storing the result back updates the tally. This single line replaces the if/else check shown in the diagram.

💡

The .get() Shortcut

You can write the accumulator pattern in two ways. The explicit way uses if char in freq: followed by an else block. The shorter way uses freq.get(char, 0) to handle both cases in one line. Both produce the same result — choose whichever feels clearer to you right now.

A complete character-frequency program looks like this in pseudocode: (1) create an empty dictionary, (2) loop over every character in the text, (3) use the accumulator pattern to update the dictionary, and (4) print or return the dictionary. The word-frequency version is almost identical — you just loop over text.split() instead of looping over the raw string.

SECTION 5

Frequency Analysis in Depth

The frequency dictionary you build can be visualized as a bar chart where each bar represents one unique character (or word) and its height represents the count. The diagram below shows the letter frequencies for the sentence "the cat sat on the mat" after converting to lowercase and removing spaces.

The letter t appears 4 times (the tallest bar), while several letters appear only once. This kind of analysis is the basis for identifying the most common characters in any text.

Notice an important pre-processing step: before counting, the text was converted to lowercase using text.lower(). Without this, Python would treat "T" and "t" as different keys. Spaces were also skipped during counting using if char != " " or by filtering with char.isalpha(). These small cleaning steps are essential for meaningful results.

Pre-processing Step	Python Method	Why It Matters
Convert to lowercase	`text.lower()`	Merges 'A' and 'a' into one key
Remove punctuation	`char.isalpha()`	Prevents commas, periods, etc. from being counted as letters
Strip extra whitespace	`text.split()`	split() handles multiple spaces and tabs automatically

Common pre-processing steps for text statistics

SECTION 6

Worked Example

Let's walk through a complete program that takes a sentence and prints its character count, word count, and letter frequency dictionary. We will use the input string "Python is fun!".

Complete Text Statistics Program

1
Step 1 — Define the Input String

Start by storing the text in a variable: text = "Python is fun!". This string has 14 characters including the space and exclamation mark.

text = "Python is fun!"
2
Step 2 — Character Count

Use the len() function to find the total number of characters. char_count = len(text) stores the result.

char_count = 14
3
Step 3 — Word Count

Split the string into a list of words with words = text.split(), which yields ["Python", "is", "fun!"]. Then word_count = len(words) gives us the word count.

word_count = 3
4
Step 4 — Build a Letter Frequency Dictionary

Create an empty dictionary: freq = {}. Then loop through each character of the lowercase version of the text. Inside the loop, skip non-letter characters using if char.isalpha(), and accumulate counts with freq[char] = freq.get(char, 0) + 1.

freq = {'p': 1, 'y': 1, 't': 1, 'h': 1, 'o': 1, 'n': 2, 'i': 1, 's': 1, 'f': 1, 'u': 1}
5
Step 5 — Print the Results

Use print() statements to display each statistic. The full code is: text = "Python is fun!" print("Characters:", len(text)) print("Words:", len(text.split())) freq = {} for char in text.lower(): if char.isalpha(): freq[char] = freq.get(char, 0) + 1 print("Frequencies:", freq)

Output: Characters: 14, Words: 3, Frequencies: {'p': 1, 'y': 1, 't': 1, 'h': 1, 'o': 1, 'n': 2, 'i': 1, 's': 1, 'f': 1, 'u': 1}

🔑

WHY THIS MATTERS

This 7-line program demonstrates three of the most common tasks in text processing. Every real-world text analysis tool — from plagiarism detectors to readability checkers — starts with exactly these operations. Master this pattern, and you can adapt it to count anything: sentence lengths, vowel percentages, unique words, and more.

SECTION 7

Comparing Different Approaches

There are several ways to count frequencies in Python. Each approach has trade-offs in terms of readability, brevity, and what you need to understand to use it. The table below compares the three most common methods.

Approach	Code Pattern	Strengths	Limitations
if/else in loop	`if char in freq: freq[char] += 1 else: freq[char] = 1`	Very explicit; easy to read and debug for beginners	Takes 4 lines instead of 1; more verbose
.get() method	`freq[char] = freq.get(char, 0) + 1`	One-liner; concise and Pythonic	Slightly harder to understand at first glance
str.count() method	`text.count("a")`	Built-in; no loop needed for a single character	Must call it separately for each character; inefficient for full analysis

Three common approaches to counting character frequency

✦

KEY TAKEAWAY

For a beginner course, the if/else loop and the .get() method are the two approaches you should master. They teach you how loops and dictionaries work together. The str.count() method is handy for quick checks but does not scale when you need to count every unique character at once.

SECTION 8

Connection to Advanced Concepts

The skills you learn here are the foundation for more powerful tools in Python and in the broader field of computer science. As you progress, you will encounter built-in libraries and data structures that automate what you are doing manually — but understanding the manual approach is crucial for knowing what those tools do under the hood.

What You Do Now	What You'll Learn Next
Build frequency dicts with loops and `.get()`	Use `collections.Counter` to do it in one line
Count words with `text.split()`	Use regular expressions (`re` module) for complex tokenization
Compute basic counts and frequencies	Calculate TF-IDF scores for search engines and document ranking
Analyze single strings	Read and analyze entire text files using `open()` and file I/O

From manual text statistics to advanced NLP

If you continue into data science or artificial intelligence, you will discover that word frequency analysis is the starting point for techniques like sentiment analysis (determining whether a review is positive or negative) and topic modeling (figuring out what a document is about). The loop-and-dictionary pattern you are practicing today is genuinely the same logic that powers these advanced applications — just at a larger scale.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

Explain in your own words why you need to convert a string to lowercase before building a letter frequency dictionary. What would happen if you skipped this step when analyzing the string "Hello World"?

Show answer

Python treats uppercase and lowercase letters as different characters because they have different character codes. If you do not call .lower() first, the dictionary would contain separate entries for "H" and "h", as well as "W" and "w". This would split the counts for what is logically the same letter, giving you inaccurate frequency data. Converting to lowercase first merges all case variants into a single key.

Question 2: BASIC CALCULATION

Given the string text = "banana", what are the values of len(text), len(text.split()), and the complete letter frequency dictionary?

Show answer

len(text) returns 6, because "banana" has six characters. len(text.split()) returns 1, because there are no spaces so the entire string is one word. The frequency dictionary is {"b": 1, "a": 3, "n": 2}. The letter 'a' appears three times, 'n' appears twice, and 'b' appears once.

Question 3: INTERMEDIATE

Write a Python program that takes the string "To be or not to be" and prints a dictionary of word frequencies (not letter frequencies). Remember to handle case sensitivity. What does the output dictionary look like?

Show answer

First convert the string to lowercase with .lower(), then split it into words. Loop over the words and use the accumulator pattern. The code would be:

text = "To be or not to be"
words = text.lower().split()
freq = {}
for word in words:
    freq[word] = freq.get(word, 0) + 1
print(freq)

The output is {"to": 2, "be": 2, "or": 1, "not": 1}. Both "To" and "to" are merged into the key "to" with a count of 2, and "be" also appears twice.

Question 4: APPLIED

A teacher wants to know which letter is most common in a student's essay. The essay text is stored in a variable called essay. Write a Python program that builds a letter frequency dictionary and then prints the letter with the highest count. Assume the essay is already stored as a string.

Show answer

Build the frequency dictionary as usual, then find the maximum. One approach:

freq = {}
for char in essay.lower():
    if char.isalpha():
        freq[char] = freq.get(char, 0) + 1

max_letter = ""
max_count = 0
for letter, count in freq.items():
    if count > max_count:
        max_count = count
        max_letter = letter
print("Most common letter:", max_letter, "appears", max_count, "times")

The first loop builds the dictionary. The second loop iterates through the dictionary's key-value pairs, keeping track of the letter with the highest count seen so far. In most English texts, 'e' will likely be the winner. You could also use max(freq, key=freq.get) as a one-liner shortcut.

Question 5: CRITICAL THINKING

Consider the string "It's a well-known fact.". If you use text.split() to count words, what list do you get, and does the count accurately reflect the number of real English words? Propose a strategy to handle contractions and hyphenated words more accurately.

Show answer

text.split() returns ["It's", "a", "well-known", "fact."] — a list of 4 items. However, "well-known" could arguably be two words, and "fact." includes a period that is not part of the word itself. The word count of 4 is reasonable for basic analysis but is imprecise. To improve accuracy, you could strip punctuation from each word using a loop: clean_words = [w.strip(".,!?;:") for w in text.split()]. For hyphenated words, you would need to decide on a rule — either treat them as one word or split at the hyphen using replace("-", " ") before splitting. There is no single "correct" answer; the right choice depends on your goal. This ambiguity is why professional NLP tools use sophisticated tokenizers.

SUMMARY

Lesson Summary

Summary

In this lesson you learned three core text statistics operations. Character count uses len(text) to find the total number of characters. Word count uses len(text.split()) to split the text at whitespace and count the resulting list. Frequency analysis uses a for loop combined with a dictionary accumulator pattern — either the if/else approach or the .get() one-liner — to count how many times each character or word appears.

Key pre-processing steps include converting to lowercase with .lower() and filtering non-alphabetic characters with .isalpha(). These techniques form the foundation for more advanced topics like natural language processing, file analysis, and data science. By mastering loops and dictionaries for text statistics, you have a reusable pattern that applies to almost any counting problem in programming.