Case Study: When Tests Lie — False Confidence from Bad Tests

The Scenario

Jordan Chen was proud of their test suite. Their student project — a grade management system for a tutoring center — had 94% code coverage and 38 passing tests. All green. Every commit. Jordan felt confident submitting the project for their software engineering course.

Then the professor ran the system with real data. It crashed in five different ways within the first three minutes.

How can a project with 38 passing tests and 94% coverage be so broken? Because the tests were bad. Not wrong — bad. They tested the wrong things, made incorrect assumptions, and gave Jordan a dangerous sense of confidence in code that didn't work.

This case study examines five testing antipatterns — common mistakes that make tests worse than useless, because they create the illusion of reliability.

Antipattern 1: The Test That Tests Nothing

Jordan's most common mistake was writing tests that executed code without verifying results:

def test_calculate_gpa():
    students = load_students("test_data.json")
    for student in students:
        calculate_gpa(student)
    # Look, no assertion!

This test runs calculate_gpa() for every student. If the function doesn't crash, the test passes. But it never checks whether the GPAs are correct. The function could return 0.0 for every student and this test would still pass.

Coverage report: 100% of calculate_gpa() is covered. The metric says everything is tested. The reality says nothing is verified.

The Fix

Every test needs at least one meaningful assertion:

def test_calculate_gpa_straight_a():
    student = {"name": "Alice", "grades": ["A", "A", "A", "A"]}
    assert calculate_gpa(student) == pytest.approx(4.0)

def test_calculate_gpa_mixed():
    student = {"name": "Bob", "grades": ["A", "B", "C"]}
    assert calculate_gpa(student) == pytest.approx(3.0)

Rule of thumb: If your test has no assert statement, it's not a test — it's a smoke test at best.

Antipattern 2: Testing the Implementation, Not the Behavior

Jordan wrote tests that were tightly coupled to how the code worked internally rather than what it was supposed to do:

def test_sort_students():
    students = [
        {"name": "Charlie", "gpa": 3.2},
        {"name": "Alice", "gpa": 3.8},
        {"name": "Bob", "gpa": 3.5},
    ]
    result = sort_students_by_gpa(students)
    # Testing the exact internal order of operations
    assert result[0] == {"name": "Alice", "gpa": 3.8}
    assert result[1] == {"name": "Bob", "gpa": 3.5}
    assert result[2] == {"name": "Charlie", "gpa": 3.2}

This test seems fine until Jordan refactored sort_students_by_gpa() to also include a "rank" field in each dict. Suddenly the test fails — not because the sorting is wrong, but because the dict now has an extra key the test didn't expect.

The Fix

Test the behavior (output is sorted by GPA, descending) rather than the exact output structure:

def test_sort_students_by_gpa_descending():
    students = [
        {"name": "Charlie", "gpa": 3.2},
        {"name": "Alice", "gpa": 3.8},
        {"name": "Bob", "gpa": 3.5},
    ]
    result = sort_students_by_gpa(students)
    gpas = [s["gpa"] for s in result]
    assert gpas == sorted(gpas, reverse=True)  # Verify descending order
    assert len(result) == len(students)  # Verify nothing was lost

Now the test verifies the contract (sorted descending, same number of students) without being brittle about internal details.

Antipattern 3: The Tautological Test

A tautological test is one that can never fail because it tests the code against itself:

def test_student_average():
    scores = [85, 90, 78, 92]
    expected = sum(scores) / len(scores)  # Same calculation as the function!
    assert calculate_average(scores) == expected

If calculate_average() has a bug — say, it accidentally uses len(scores) - 1 — this test would still pass, because expected is computed using the same (correct) formula, not the (buggy) function. The test is comparing the function to independent logic, which is good — but only if the "independent logic" is actually independent.

The problem is subtler here: sum(scores) / len(scores) happens to be the correct formula. But the test would be more resilient if it used a hardcoded expected value:

The Fix

def test_student_average():
    scores = [85, 90, 78, 92]
    assert calculate_average(scores) == pytest.approx(86.25)  # Hardcoded expected value

Hardcoded expected values are computed by hand (or with a calculator) — they're truly independent of the code being tested.

Antipattern 4: Tests That Depend on Each Other

Jordan's test file had tests that relied on state left behind by previous tests:

students_db = []  # Module-level shared state

def test_add_student():
    students_db.append({"name": "Alice", "gpa": 3.8})
    assert len(students_db) == 1

def test_find_student():
    # Assumes test_add_student ran first and added Alice
    result = find_in_list(students_db, "Alice")
    assert result["name"] == "Alice"

def test_remove_student():
    # Assumes both previous tests ran and Alice is in the list
    students_db.remove({"name": "Alice", "gpa": 3.8})
    assert len(students_db) == 0

These tests pass when run together in order. But: - Run test_find_student alone? It fails (empty list). - Run tests in random order (which pytest can do)? Failures everywhere. - test_add_student fails? Both subsequent tests fail too, but for the wrong reason.

The Fix

Each test creates its own data using fixtures:

@pytest.fixture
def students_db():
    return [{"name": "Alice", "gpa": 3.8}]

def test_add_student():
    db = []
    db.append({"name": "Alice", "gpa": 3.8})
    assert len(db) == 1

def test_find_student(students_db):
    result = find_in_list(students_db, "Alice")
    assert result["name"] == "Alice"

def test_remove_student(students_db):
    students_db.remove({"name": "Alice", "gpa": 3.8})
    assert len(students_db) == 0

Rule of thumb: Every test should pass regardless of which other tests have run (or haven't run) before it.

Antipattern 5: Testing Only the Happy Path

Jordan's test suite had 38 tests. Thirty-five tested normal inputs with expected data. Only three tested anything unusual. None tested: - Empty input - Invalid data types - Extremely large datasets - Missing required fields - Concurrent modifications - Unicode characters in names

The first real dataset the professor used had a student named "María" (with an accent) and another with the last name "O'Brien." Both caused crashes because the code assumed ASCII-only names. A student with no grades caused a division-by-zero error. A student record missing the "name" field caused a KeyError.

The Fix

For every test of normal behavior, write at least one test of abnormal behavior:

# Happy path
def test_gpa_normal():
    assert calculate_gpa({"name": "Alice", "grades": ["A", "B"]}) == pytest.approx(3.5)

# Edge cases
def test_gpa_no_grades():
    with pytest.raises(ValueError):
        calculate_gpa({"name": "Alice", "grades": []})

def test_gpa_unicode_name():
    result = calculate_gpa({"name": "María", "grades": ["A"]})
    assert result == pytest.approx(4.0)

def test_gpa_missing_name():
    with pytest.raises(KeyError):
        calculate_gpa({"grades": ["A", "B"]})

The Lesson

Jordan rewrote their test suite. The new version had 52 tests, but they were fundamentally different: - Every test had meaningful assertions - Tests verified behavior, not implementation details - Expected values were hardcoded, not computed - Each test was independent - Edge cases outnumbered happy-path tests

Coverage dropped from 94% to 87%. But the new suite caught real bugs. The old suite just made Jordan feel safe.

The most dangerous test suite is one that passes when it shouldn't. It's worse than having no tests at all, because no tests at least leave you aware of your uncertainty. Bad tests replace uncertainty with false confidence.

Antipattern Summary Table

Discussion Questions

1. Jordan's test suite had 94% coverage and 38 passing tests. Why didn't these metrics reveal the quality problems? What additional metrics or practices could have caught the issues earlier?

2. "The most dangerous test suite is one that passes when it shouldn't." Explain why false confidence from bad tests can be worse than having no tests at all. What real-world situations have similar dynamics (feeling safe when you're not)?

3. Pick one of the five antipatterns and describe a situation where you might accidentally fall into it. What habit or practice would help you avoid it?

4. Connection to Section 13.5 (TDD): How would TDD have prevented Antipattern 1 (no assertions)? Think about the Red-Green-Refactor cycle — at which step would the problem become obvious?

5. Review your own code from a previous chapter. Do any of these antipatterns apply to how you tested it (even manually)? What would a proper test suite look like?

Mini-Project

Take Jordan's original calculate_gpa() scenario and build it properly:

1. Write calculate_gpa(student) where a student dict has "name" (str) and "grades" (list of letter grades: A=4.0, B=3.0, C=2.0, D=1.0, F=0.0)
2. Write a test suite that avoids all five antipatterns
3. Include at least 3 happy-path tests and at least 5 edge/error case tests
4. Aim for 90%+ coverage, but explain which lines you chose not to cover and why
5. Have a classmate review your tests and try to find a case your tests miss