Deterministic vs. Non-Deterministic Tests — Understand deterministic vs non-deterministic failures (intro) - Python

SECTION 1

Historical Context & Motivation

Software testing has been a part of programming almost since the very beginning. In the early days of computing, programmers would run their programs and manually check the output to see if anything went wrong. As software grew more complex, teams realized they needed automated ways to verify their code. This led to the rise of automated testing, where scripts check the program's behavior without human intervention.

However, as test suites grew to hundreds or thousands of tests, developers noticed a frustrating pattern: some tests would pass one minute and fail the next, even though nobody had changed the code. These unpredictable failures earned the nickname flaky tests. Understanding why some tests behave consistently while others do not became one of the most important challenges in software quality.

1960s • Early Manual Testing

Programmers at NASA and universities manually checked outputs of their programs on mainframe computers. Testing was slow, tedious, and entirely human-driven.

1998 • JUnit and Automated Test Frameworks

Kent Beck and Erich Gamma released JUnit for Java, inspiring frameworks like Python's unittest and later pytest. Automated testing became mainstream.

2008 • pytest Gains Popularity

The pytest framework rose in the Python community, making it easy to write tests with simple assert statements and rich error messages.

2016 • Google's Flaky Test Study

Google published research showing that roughly 1 in 7 of their tests exhibited flaky (non-deterministic) behavior. The study highlighted the cost of ignoring flaky tests at scale.

So the central question is: why do some tests always give you the same result when nothing has changed, while others seem to flip between pass and fail randomly? The answer lies in understanding the difference between deterministic and non-deterministic tests.

SECTION 2

Core Principles & Definitions

Before diving into examples, let's establish the key ideas that separate reliable tests from unreliable ones. A deterministic test is one that always produces the same result — pass or fail — when run against the same code. If you run it once and it passes, running it a hundred more times on the same code will also pass. On the other hand, a non-deterministic test (also called a flaky test) might pass on one run and fail on the next, even though the source code hasn't changed at all.

Deterministic Test

Always produces the same outcome (pass or fail) for the same code. The result depends only on the code being tested, with no hidden variables.

Non-Deterministic Test

May produce different outcomes on different runs, even when code hasn't changed. External factors like timing, randomness, or network state influence the result.

Flaky Test

A common name for a non-deterministic test. It 'flakes out' by passing sometimes and failing other times, making developers lose trust in the test suite.

External Dependency

Any factor outside the code being tested — network connections, file systems, time of day, random number generators — that can introduce non-determinism.

Test Isolation

The practice of making each test independent so that one test's behavior doesn't affect another's outcome. Good isolation helps keep tests deterministic.

✦

KEY TAKEAWAY

Think of a deterministic test like a vending machine: put in the same dollar bill and press the same button, you always get the same snack. A non-deterministic test is like a vending machine with a loose connection — sometimes your snack drops, sometimes it gets stuck, and you can never be sure which will happen next.

SECTION 3

Visual Explanation

The diagram below contrasts what happens when you run a deterministic test versus a non-deterministic test multiple times. On the left, every run produces the same outcome. On the right, the outcome varies unpredictably from run to run.

Left: a deterministic test produces the same result every time the code is unchanged. Right: a non-deterministic (flaky) test yields inconsistent results despite no code changes.

Notice that in both panels the underlying code is exactly the same. The deterministic test on the left shows a clean green column — every run passes. The non-deterministic test on the right shows a mix of green passes and red failures. This inconsistency is what makes flaky tests so dangerous: you can't tell whether a failure means your code is broken or whether the test itself is unreliable.

SECTION 4

How Non-Determinism Sneaks Into Tests

Non-determinism doesn't appear out of thin air. It comes from specific, identifiable sources in your code or test environment. Understanding these sources is the first step toward writing reliable tests. Let's walk through the most common culprits in Python.

Source 1 — Randomness

If your code uses Python's random module and your test doesn't set a fixed seed (a starting value for the random number generator), the output will change every time. For example, random.choice(["heads", "tails"]) gives different results on different runs. A test that checks for a specific result without controlling the seed will be flaky.

Source 2 — Time Dependence

Tests that depend on the current time or date can behave differently at different moments. Imagine a function that returns "Good morning" before noon and "Good afternoon" after noon. A test that expects "Good morning" will pass in the morning but fail in the afternoon. The code is correct, but the test is time-dependent.

Source 3 — Network & External Services

If a test calls an external API or reads from a website, the result depends on whether the server is up, how fast the network is, and what data the server returns. Any of these factors can change between test runs, making the test non-deterministic.

Source 4 — Shared State & Test Order

When tests share data — such as writing to the same file or modifying the same global variable — the order in which tests run matters. If Test A creates a file and Test B reads that file, but Test B runs before Test A, Test B will fail. Some test runners randomize test order by default, which exposes this kind of hidden dependency.

Source 5 — Concurrency & Timing

Code that uses threads or asynchronous operations can produce different results depending on how fast each operation completes. A race condition occurs when two operations compete to finish first, and the test result depends on the winner. This is a common cause of flakiness in more advanced programs.

💡

Quick Tip

Whenever you write a test, ask yourself: "Does this test depend on anything outside my code?" If the answer is yes, that's a potential source of non-determinism. You'll want to control or eliminate that dependency.

SECTION 5

Classifying Test Failures

When a test fails, developers need to quickly determine whether it's a deterministic failure (a real bug in the code) or a non-deterministic failure (a problem with the test itself). The diagram below shows a decision flowchart for classifying failures.

A simple flowchart for classifying failures: re-run the test without changing code. If it consistently fails, the bug is in your code. If it sometimes passes, the test itself is flaky.

Feature	Deterministic Failure	Non-Deterministic Failure
Reproducibility	Fails every time on the same code	Fails only sometimes; passes on re-run
Root Cause	Bug in the actual code logic	External dependency, timing, or shared state
Action to Fix	Debug and fix the source code	Redesign the test to eliminate external factors
Trustworthiness	High — you can trust the failure signal	Low — developers start ignoring failures
Example	`assert add(2, 3) == 6` (wrong expected value)	Test checks current time or network response

Key differences between deterministic and non-deterministic test failures

SECTION 6

Worked Example — Identifying and Fixing a Flaky Test

Let's walk through a realistic scenario where a Python test is non-deterministic, identify the cause, and fix it to make it deterministic.

Fixing a Flaky Random-Dependent Test

1
Step 1 — Read the Original Code

Suppose we have a function that generates a random password. The code uses random.choices() to pick characters from a set of letters and digits:
2
Step 2 — Read the Flaky Test

A developer wrote this test: def test_generate_password(): result = generate_password(8) assert result == "aB3xK9mZ" This test checks if the password is exactly "aB3xK9mZ". Since the function uses random characters, the generated password will be different nearly every run. This test will occasionally pass by pure luck, but almost always fail.

Non-deterministic — the test depends on random output
3
Step 3 — Identify the Root Cause

The test is comparing the output to a specific string, but the function produces a random result each time. The root cause of flakiness is the uncontrolled randomness in the function.
4
Step 4 — Choose a Fix Strategy

We have two good options. Option A: test the properties of the output (length and character set) instead of the exact string. Option B: use random.seed() to make the random output predictable in the test. Option A is usually better because it tests what truly matters.
5
Step 5 — Write the Fixed Deterministic Test

Here is the improved test using Option A: import string def test_generate_password(): result = generate_password(8) assert len(result) == 8 valid_chars = string.ascii_letters + string.digits assert all(c in valid_chars for c in result) This test checks that the password has the correct length and uses only valid characters. It will pass every time the function works correctly, regardless of which specific characters are chosen.

Deterministic — the test checks properties, not exact random output

SECTION 7

Strengths & Limitations of Each Approach

Both deterministic and non-deterministic tests exist in the real world, and understanding the tradeoffs helps you decide how to handle them. The table below summarizes the key pros and cons.

Aspect	Deterministic Tests	Non-Deterministic Tests
Reliability	Always reliable — a fail always means a real problem	Unreliable — a fail might mean nothing
Developer Trust	Developers pay attention to failures	Developers learn to ignore failures ('cry wolf' effect)
Debugging Effort	Easy — reproduce the bug every time	Hard — can't always reproduce the failure
Design Effort	Requires careful design to isolate tests from external factors	Quick to write but incurs long-term cost
CI/CD Impact	Builds are green or red — clear signal	Teams re-run builds hoping for green, wasting time and resources

Comparing deterministic and non-deterministic tests across key dimensions

✦

KEY TAKEAWAY

Flaky tests are like a car alarm that goes off randomly. At first, people rush to check. After a few false alarms, everyone ignores it — even when someone is actually breaking into the car. That's why non-deterministic tests are dangerous: they train your team to ignore real warnings.

SECTION 8

Connection to Advanced Testing Practices

The concepts you've learned here form the foundation of more advanced testing practices used by professional software teams. As you move into AP Computer Science or college-level courses, you'll encounter techniques specifically designed to keep tests deterministic.

Concept from This Lesson	Advanced Technique
Controlling randomness with seeds	Property-based testing (e.g., Hypothesis library) — generates random inputs systematically and records the seed for reproducibility
Avoiding network dependencies	Mocking and patching (e.g., unittest.mock) — replaces real network calls with fake, predictable responses
Eliminating time dependence	Freezing time (e.g., freezegun library) — lets you set a fixed date/time for your tests
Preventing shared state issues	Test fixtures and setup/teardown — create a clean environment before each test and clean up after
Detecting flaky tests	Flaky test detection tools — CI systems like GitHub Actions can automatically re-run and flag tests that behave inconsistently

How introductory concepts connect to professional testing tools

The key idea that carries forward is this: a good test controls every variable that could change the outcome. As the systems you test grow more complex, the tools get more sophisticated, but the core principle remains the same. Your goal is always to make the test's result depend on only one thing: whether the code is correct.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

In your own words, explain the difference between a deterministic test and a non-deterministic test. Why is it important for developers to know which type a failing test is?

Show answer

A deterministic test always gives the same result (pass or fail) when run against the same code, while a non-deterministic test may give different results on different runs even with identical code. Knowing the difference matters because a deterministic failure reliably signals a bug in the code, so you should go fix the code. A non-deterministic failure may signal a problem with the test's design — perhaps it depends on timing, randomness, or a network resource. If developers don't distinguish between the two, they might waste time chasing phantom bugs or, worse, start ignoring all test failures.

Question 2: BASIC CALCULATION

Consider this test:

import random

def test_coin_flip():
    result = random.choice(["heads", "tails"])
    assert result == "heads"

Is this test deterministic or non-deterministic? Explain why, and suggest one way to fix it.

Show answer

This test is non-deterministic because random.choice() returns a different value on different runs. Roughly half the time the test will pass (when "heads" is chosen), and half the time it will fail (when "tails" is chosen). To fix it, you could set a seed at the beginning of the test: random.seed(42) makes the random number generator produce the same sequence every time. Alternatively, you could assert that the result is in the set {"heads", "tails"}, testing the property rather than a specific outcome.

Question 3: INTERMEDIATE

A test suite has three tests. Test A writes a value to a global list. Test B reads from that same global list and checks its length. Test C clears the list. When the tests run in order A → B → C, everything passes. But when the test runner randomizes the order, Test B sometimes fails. Identify the source of non-determinism and describe how you would fix it.

Show answer

The source of non-determinism is shared state — all three tests depend on the same global list, and the outcome of Test B depends on whether Test A has run before it. If the order is B → A → C, Test B reads an empty list and fails. The fix is test isolation: each test should set up its own data at the start and clean up after itself. In pytest, you could use a fixture that creates a fresh list for each test. For example, you could add

@pytest.fixture
def fresh_list():
    return []

and pass it as an argument to each test function. This way, no test depends on another test's side effects, and the order no longer matters.

Question 4: APPLIED

You're building a weather app in Python that calls an external API to get the current temperature. Your test looks like this:

def test_get_weather():
    temp = get_current_temp("New York")
    assert isinstance(temp, float)
    assert -50 < temp < 150

Even though this test checks a reasonable range, it can still be non-deterministic. Explain why, and describe a strategy to make it deterministic.

Show answer

This test is non-deterministic because it depends on an external API call. If the weather API server is down, the function might raise an exception instead of returning a float, causing the test to fail. If the network is slow, the function might time out. Even if the API is working, the returned temperature changes over time. To make this test deterministic, you could use mocking to replace the real API call with a fake function that always returns a known value, such as 72.5. In Python, you can use unittest.mock.patch to substitute the API call. This ensures the test always receives the same input, making the outcome purely dependent on your code's logic, not the outside world.

Question 5: CRITICAL THINKING

A team decides to "solve" their flaky test problem by simply re-running every failed test three times and only counting it as a real failure if it fails all three times. Evaluate this strategy. What are the benefits and what are the risks? Could this approach ever mask a real bug?

Show answer

This retry strategy has a practical benefit: it reduces the number of false alarms from flaky tests, which means developers are interrupted less often by meaningless failures. However, it comes with significant risks. First, it slows down the test suite because every flaky failure triggers multiple re-runs. Second, and more dangerously, it can mask real bugs. Consider a race condition in the code that causes a failure roughly one out of three times. With the retry strategy, the test would need to fail three times in a row to be flagged — a probability of roughly 1/27 (about 3.7%). The team might never notice this real, intermittent bug. The better long-term strategy is to fix the root cause of flakiness rather than papering over it with retries. Retries can be a short-term workaround but should always be paired with a plan to investigate and fix the underlying non-determinism.

SUMMARY

Lesson Summary

Summary

In this lesson, you learned the critical distinction between deterministic tests and non-deterministic (flaky) tests. A deterministic test always produces the same result for the same code, while a non-deterministic test can flip between passing and failing. The five main sources of non-determinism are uncontrolled randomness, time dependence, network and external services, shared state between tests, and concurrency and race conditions.

To classify a failure, re-run the test without changing code: consistent failure points to a real bug, while intermittent failure points to a test design problem. The best strategy for writing reliable tests is to isolate each test from external factors — control randomness with seeds, avoid live network calls, test properties instead of exact values, and give each test its own clean state. These habits will serve you well as you move toward more advanced topics like mocking, fixtures, and continuous integration.