Historical Context & Motivation
Software testing has been a part of programming almost since the very beginning. In the early days of computing, programmers would run their programs and manually check the output to see if anything went wrong. As software grew more complex, teams realized they needed automated ways to verify their code. This led to the rise of automated testing, where scripts check the program's behavior without human intervention.
However, as test suites grew to hundreds or thousands of tests, developers noticed a frustrating pattern: some tests would pass one minute and fail the next, even though nobody had changed the code. These unpredictable failures earned the nickname flaky tests. Understanding why some tests behave consistently while others do not became one of the most important challenges in software quality.
So the central question is: why do some tests always give you the same result when nothing has changed, while others seem to flip between pass and fail randomly? The answer lies in understanding the difference between deterministic and non-deterministic tests.
Core Principles & Definitions
Before diving into examples, let's establish the key ideas that separate reliable tests from unreliable ones. A deterministic test is one that always produces the same result — pass or fail — when run against the same code. If you run it once and it passes, running it a hundred more times on the same code will also pass. On the other hand, a non-deterministic test (also called a flaky test) might pass on one run and fail on the next, even though the source code hasn't changed at all.
Deterministic Test
Non-Deterministic Test
Flaky Test
External Dependency
Test Isolation
Visual Explanation
The diagram below contrasts what happens when you run a deterministic test versus a non-deterministic test multiple times. On the left, every run produces the same outcome. On the right, the outcome varies unpredictably from run to run.
Notice that in both panels the underlying code is exactly the same. The deterministic test on the left shows a clean green column — every run passes. The non-deterministic test on the right shows a mix of green passes and red failures. This inconsistency is what makes flaky tests so dangerous: you can't tell whether a failure means your code is broken or whether the test itself is unreliable.
How Non-Determinism Sneaks Into Tests
Non-determinism doesn't appear out of thin air. It comes from specific, identifiable sources in your code or test environment. Understanding these sources is the first step toward writing reliable tests. Let's walk through the most common culprits in Python.
Source 1 — Randomness
If your code uses Python's random module and your test doesn't set a fixed seed (a starting value for the random number generator), the output will change every time. For example, random.choice(["heads", "tails"]) gives different results on different runs. A test that checks for a specific result without controlling the seed will be flaky.
Source 2 — Time Dependence
Tests that depend on the current time or date can behave differently at different moments. Imagine a function that returns "Good morning" before noon and "Good afternoon" after noon. A test that expects "Good morning" will pass in the morning but fail in the afternoon. The code is correct, but the test is time-dependent.
Source 3 — Network & External Services
If a test calls an external API or reads from a website, the result depends on whether the server is up, how fast the network is, and what data the server returns. Any of these factors can change between test runs, making the test non-deterministic.
Source 4 — Shared State & Test Order
When tests share data — such as writing to the same file or modifying the same global variable — the order in which tests run matters. If Test A creates a file and Test B reads that file, but Test B runs before Test A, Test B will fail. Some test runners randomize test order by default, which exposes this kind of hidden dependency.
Source 5 — Concurrency & Timing
Code that uses threads or asynchronous operations can produce different results depending on how fast each operation completes. A race condition occurs when two operations compete to finish first, and the test result depends on the winner. This is a common cause of flakiness in more advanced programs.
Classifying Test Failures
When a test fails, developers need to quickly determine whether it's a deterministic failure (a real bug in the code) or a non-deterministic failure (a problem with the test itself). The diagram below shows a decision flowchart for classifying failures.
| Feature | Deterministic Failure | Non-Deterministic Failure |
|---|---|---|
| Reproducibility | Fails every time on the same code | Fails only sometimes; passes on re-run |
| Root Cause | Bug in the actual code logic | External dependency, timing, or shared state |
| Action to Fix | Debug and fix the source code | Redesign the test to eliminate external factors |
| Trustworthiness | High — you can trust the failure signal | Low — developers start ignoring failures |
| Example | assert add(2, 3) == 6 (wrong expected value) | Test checks current time or network response |
Worked Example — Identifying and Fixing a Flaky Test
Let's walk through a realistic scenario where a Python test is non-deterministic, identify the cause, and fix it to make it deterministic.
random.choices() to pick characters from a set of letters and digits:def test_generate_password():
result = generate_password(8)
assert result == "aB3xK9mZ"
This test checks if the password is exactly "aB3xK9mZ". Since the function uses random characters, the generated password will be different nearly every run. This test will occasionally pass by pure luck, but almost always fail.random.seed() to make the random output predictable in the test. Option A is usually better because it tests what truly matters.import string
def test_generate_password():
result = generate_password(8)
assert len(result) == 8
valid_chars = string.ascii_letters + string.digits
assert all(c in valid_chars for c in result)
This test checks that the password has the correct length and uses only valid characters. It will pass every time the function works correctly, regardless of which specific characters are chosen.Strengths & Limitations of Each Approach
Both deterministic and non-deterministic tests exist in the real world, and understanding the tradeoffs helps you decide how to handle them. The table below summarizes the key pros and cons.
| Aspect | Deterministic Tests | Non-Deterministic Tests |
|---|---|---|
| Reliability | Always reliable — a fail always means a real problem | Unreliable — a fail might mean nothing |
| Developer Trust | Developers pay attention to failures | Developers learn to ignore failures ('cry wolf' effect) |
| Debugging Effort | Easy — reproduce the bug every time | Hard — can't always reproduce the failure |
| Design Effort | Requires careful design to isolate tests from external factors | Quick to write but incurs long-term cost |
| CI/CD Impact | Builds are green or red — clear signal | Teams re-run builds hoping for green, wasting time and resources |
Connection to Advanced Testing Practices
The concepts you've learned here form the foundation of more advanced testing practices used by professional software teams. As you move into AP Computer Science or college-level courses, you'll encounter techniques specifically designed to keep tests deterministic.
| Concept from This Lesson | Advanced Technique |
|---|---|
| Controlling randomness with seeds | Property-based testing (e.g., Hypothesis library) — generates random inputs systematically and records the seed for reproducibility |
| Avoiding network dependencies | Mocking and patching (e.g., unittest.mock) — replaces real network calls with fake, predictable responses |
| Eliminating time dependence | Freezing time (e.g., freezegun library) — lets you set a fixed date/time for your tests |
| Preventing shared state issues | Test fixtures and setup/teardown — create a clean environment before each test and clean up after |
| Detecting flaky tests | Flaky test detection tools — CI systems like GitHub Actions can automatically re-run and flag tests that behave inconsistently |
The key idea that carries forward is this: a good test controls every variable that could change the outcome. As the systems you test grow more complex, the tools get more sophisticated, but the core principle remains the same. Your goal is always to make the test's result depend on only one thing: whether the code is correct.
Practice Problems
import random
def test_coin_flip():
result = random.choice(["heads", "tails"])
assert result == "heads"
Is this test deterministic or non-deterministic? Explain why, and suggest one way to fix it.def test_get_weather():
temp = get_current_temp("New York")
assert isinstance(temp, float)
assert -50 < temp < 150
Even though this test checks a reasonable range, it can still be non-deterministic. Explain why, and describe a strategy to make it deterministic.Lesson Summary
In this lesson, you learned the critical distinction between deterministic tests and non-deterministic (flaky) tests. A deterministic test always produces the same result for the same code, while a non-deterministic test can flip between passing and failing. The five main sources of non-determinism are uncontrolled randomness, time dependence, network and external services, shared state between tests, and concurrency and race conditions.
To classify a failure, re-run the test without changing code: consistent failure points to a real bug, while intermittent failure points to a test design problem. The best strategy for writing reliable tests is to isolate each test from external factors — control randomness with seeds, avoid live network calls, test properties instead of exact values, and give each test its own clean state. These habits will serve you well as you move toward more advanced topics like mocking, fixtures, and continuous integration.