Chapter 22: Regular Expressions: Pattern Matching Power

"Some people, when confronted with a problem, think 'I know, I'll use regular expressions.' Now they have two problems." — Jamie Zawinski, 1997

Chapter Overview

You've been processing text since Chapter 7. You know split(), strip(), replace(), find(), startswith(), and endswith(). Those string methods are workhorses — and for simple tasks, they're all you need.

But real-world text isn't simple.

Consider these problems: you need to find every phone number in a 10,000-line customer database, but the numbers appear as (555) 867-5309, 555-867-5309, 555.867.5309, and 5558675309. You need to pull dates out of log entries where the format shifts between 2025-03-14, 03/14/2025, and March 14, 2025. You need to validate that an email address is at least structurally plausible before sending a confirmation message.

String methods can handle one format. When the problem is "find a pattern that could appear in many shapes," you need a different tool.

That tool is the regular expression — regex for short. A regex is a mini-language for describing text patterns. Instead of saying "find the exact string 555-867-5309," you say "find three digits, then a separator, then three digits, then a separator, then four digits." The regex engine handles the rest.

Regular expressions appear everywhere in professional software. Log analysis, data cleaning, input validation, search-and-replace, web scraping, bioinformatics, network security — any domain where you process text at scale uses regex. Python's re module puts that power directly in your hands.

Fair warning: regex has a learning curve. The syntax is dense, and a complicated pattern can look like a cat walked across the keyboard. But once you learn to read it, regex becomes one of the most productive tools in your kit. A single line of regex can replace twenty lines of manual string parsing — and it'll handle edge cases you'd never think to code by hand.

In this chapter, you will learn to: - Write regex patterns for common text matching tasks - Use the re module functions: search, match, findall, sub, and split - Apply character classes, quantifiers, groups, and anchors to build precise patterns - Extract structured data from unstructured text using capture groups - Distinguish greedy from lazy matching and know when each is appropriate - Know when regex is the right tool — and when plain string methods are enough

Spaced Review: This chapter builds directly on Chapter 7 (string methods, raw strings, immutability) and draws on Chapter 10 (reading files line by line). You'll also see bridging connections to Chapter 21 (data processing pipelines).

Fast Track: If you're comfortable with basic regex syntax from another language, skim 22.1-22.3 and jump to 22.6 (groups).

Deep Dive: The case studies explore log file analysis with regex and how search engines parse user queries.

22.1 Why Regex? (Text Data Is Messy)

Let's start with a real problem. Dr. Patel — the biology researcher you first met in Chapter 1 — works with FASTA files every day. Alongside her DNA sequences, she downloads FASTA records for the proteins those genes encode from UniProt, the big public protein database. Each record starts with a header line like these (trimmed — real UniProt headers continue with more KEY=value fields):

>sp|P04637|P53_HUMAN Cellular tumor antigen p53 OS=Homo sapiens
>sp|Q9Y6K1|DNM3A_HUMAN DNA (cytosine-5)-methyltransferase 3A OS=Homo sapiens
>sp|P38398|BRCA1_HUMAN Breast cancer type 1 susceptibility protein OS=Homo sapiens

Dr. Patel needs to extract just the entry names — UniProt's short labels: P53_HUMAN, DNM3A_HUMAN, BRCA1_HUMAN. They always appear between the second | character and the first space after it.

With string methods, you'd write something like:

def extract_entry_name(header: str) -> str:
    """Extract the entry name from a UniProt FASTA header — the hard way."""
    # Find the second pipe character
    first_pipe = header.index("|")
    second_pipe = header.index("|", first_pipe + 1)
    # Find the space after the entry name
    space_after = header.index(" ", second_pipe + 1)
    return header[second_pipe + 1 : space_after]

That works, but it's fragile. What if a header has no space after the entry name? (index() raises a ValueError.) What if the format changes slightly? And what if you need to also extract the organism name (Homo sapiens) and the accession number (P04637)? You'd need more index() calls, more slicing, more variables — and the code would be hard to read and harder to maintain.

With regex, the same extraction looks like this:

import re

header = ">sp|P04637|P53_HUMAN Cellular tumor antigen p53 OS=Homo sapiens"
pattern = r">(\w+)\|(\w+)\|(\w+)\s+(.+?)\s+OS=(.+)"
match = re.search(pattern, header)
if match:
    db, accession, entry_name, description, organism = match.groups()
    print(entry_name, "|", organism)  # P53_HUMAN | Homo sapiens

One line of pattern, one line to extract five fields. And if the format changes, you update the pattern — not an entire parsing function. (Case in point: on an untrimmed UniProt header, the final (.+) would also swallow the trailing OX=9606 GN=TP53 ... fields. Ending the pattern with OS=(.+?)\s+OX= instead fixes it, with no other code touched.) Don't worry if that pattern looks like line noise right now; by Section 22.8 you'll be able to read every piece of it.

Let's build up to that level of regex fluency step by step.

What Is a Regular Expression?

A regular expression (or regex, or regexp) is a sequence of characters that defines a search pattern. You write a pattern, and the regex engine scans through text looking for substrings that match.

At its simplest, a regex is just a literal string:

import re

text = "The cat sat on the mat."
result = re.search(r"cat", text)
print(result)  # <re.Match object; span=(4, 7), match='cat'>

The pattern cat matches the literal characters c, a, t in sequence. But regex gets its power from metacharacters — characters with special meaning that let you describe classes of strings rather than one specific string.

Raw Strings: The r Prefix

You'll notice we write regex patterns with the r prefix: r"cat" instead of "cat". This creates a raw string — one where backslashes are treated as literal characters, not escape sequences.

This matters because regex uses backslashes extensively, and so do ordinary Python strings. Remember from Chapter 7 that "\n" is one newline character, not a backslash and an n. Python processes those escape sequences before the regex engine ever sees your pattern. Some regex codes collide with real Python escapes: \b means "word boundary" to the regex engine but "backspace" to Python. Others, like \d ("any digit"), aren't Python escapes at all. Python passes them through unchanged, but since Python 3.12 it prints a SyntaxWarning: invalid escape sequence '\d', and a future version will make it an error.

The raw string r"\d" sidesteps all of this: it hands the literal characters \ and d to the regex engine, which is exactly what you want.

# Without raw string — Python turns each \b into a backspace character
pattern_bad = "\bword\b"
print(len(pattern_bad))   # 6  (backspace + "word" + backspace)

# With raw string — \b reaches the regex engine as a word boundary
pattern_good = r"\bword\b"
print(len(pattern_good))  # 8  (\, b, w, o, r, d, \, b)

Best Practice: Always use raw strings for regex patterns. Always. Even when the pattern doesn't contain backslashes — it's a habit that prevents subtle bugs.

22.2 The re Module: search, match, findall

Python's re module is part of the standard library — no installation needed. It provides several functions for working with regex patterns. Let's meet the three most common.

re.search() — Find the First Match Anywhere

re.search(pattern, string) scans the entire string and returns a match object for the first occurrence of the pattern, or None if no match is found.

import re

log_line = "2025-03-14 08:23:17 ERROR Database connection timeout"

# Search for a date pattern (we'll refine this later)
result = re.search(r"\d{4}-\d{2}-\d{2}", log_line)
if result:
    print(f"Found date: {result.group()}")   # Found date: 2025-03-14
    print(f"Position: {result.span()}")       # Position: (0, 10)

The match object gives you: - .group() — the actual text that matched - .span() — the start and end positions (as a tuple) - .start() and .end() — the individual positions

re.match() — Match at the Beginning Only

re.match(pattern, string) only checks whether the pattern matches at the start of the string. It won't find matches in the middle.

log_line = "2025-03-14 08:23:17 ERROR Database connection timeout"

# match() works here — the date IS at the start
result = re.match(r"\d{4}-\d{2}-\d{2}", log_line)
print(result.group())  # 2025-03-14

# match() fails here — "ERROR" isn't at the start
result = re.match(r"ERROR", log_line)
print(result)  # None

Pitfall: A common source of confusion. re.match() does NOT check whether the pattern matches the whole string — it checks whether the pattern matches at the start. If you need to match the entire string, use re.fullmatch().

re.findall() — Find All Matches

re.findall(pattern, string) returns a list of all non-overlapping matches. This is the function you'll reach for most often.

text = "Call 555-1234 or 555-5678. Fax: 555-9999."

# Find all phone-number-like patterns
numbers = re.findall(r"\d{3}-\d{4}", text)
print(numbers)  # ['555-1234', '555-5678', '555-9999']

No match objects — just a clean list of strings. When your pattern contains groups (Section 22.6), findall() returns the group contents instead.

If you need every match and the details a match object carries (like positions), use re.finditer(). It hands you one match object at a time, ready for a for loop:

for m in re.finditer(r"\d{3}-\d{4}", text):
    print(m.group(), m.span())
# 555-1234 (5, 13)
# 555-5678 (17, 25)
# 555-9999 (32, 40)

re.split() — Split on a Pattern

Like str.split(), but the delimiter is a regex pattern:

text = "one,two;  three   four"

# Split on any combination of comma, semicolon, or whitespace
parts = re.split(r"[,;\s]+", text)
print(parts)  # ['one', 'two', 'three', 'four']

str.split() can't do this in one step: it takes a single delimiter string (or, with no argument, splits on runs of whitespace), but never "a comma or a semicolon or spaces."

Check Your Understanding #1

What's the difference between re.search(r"hello", text) and re.match(r"hello", text) when text = "Say hello!"?

22.3 Character Classes: [abc], \d, \w, \s

So far our patterns have been literal characters and \d (which we haven't formally explained). Let's fix that.

A character class matches any single character from a defined set. You write it inside square brackets.

import re

# Match any vowel
text = "Hello, World!"
vowels = re.findall(r"[aeiouAEIOU]", text)
print(vowels)  # ['e', 'o', 'o']

Ranges in Character Classes

Use a hyphen to specify a range:

# Any lowercase letter
re.findall(r"[a-z]", "Hello 123")       # ['e', 'l', 'l', 'o']

# Any digit
re.findall(r"[0-9]", "Hello 123")       # ['1', '2', '3']

# Any letter (upper or lower) or digit
re.findall(r"[a-zA-Z0-9]", "Hi! 42")   # ['H', 'i', '4', '2']

Negated Character Classes

A caret ^ inside brackets means "NOT these characters":

# Any character that is NOT a digit
re.findall(r"[^0-9]", "Hello 123")  # ['H', 'e', 'l', 'l', 'o', ' ']

Shorthand Character Classes

Typing [0-9] every time gets old. Regex provides shorthand:

The "Equivalent" column is exactly right for plain English (ASCII) text. Python strings are Unicode, though, so the shorthands are a bit more generous: \w also matches accented letters like é and ñ, \d matches digits from other writing systems, and \s matches extra whitespace characters such as the non-breaking space. Usually that's what you want — re.findall(r"\w+", "café") gives ['café'], not ['caf']. If you need the strict ASCII meaning, pass the re.ASCII flag or spell out the class, like [0-9].

The dot . is the wildcard — it matches any single character except \n. It's powerful but dangerously broad. Use a specific character class when you can.

# Find all "words" (sequences of word characters)
text = "Dr. Patel has 3 gene files!"
words = re.findall(r"\w+", text)
print(words)  # ['Dr', 'Patel', 'has', '3', 'gene', 'files']

# Find all whitespace-separated tokens
tokens = re.findall(r"\S+", text)
print(tokens)  # ['Dr.', 'Patel', 'has', '3', 'gene', 'files!']

Matching a Literal Metacharacter

What if you want to match an actual dot, parenthesis, or dollar sign? Characters like . ^ $ * + ? { } [ ] \ | ( ) all have special meanings, so you escape them with a backslash: \. matches a literal period, \( a literal opening parenthesis, \$ a literal dollar sign.

# Unescaped: . means "any character", so "3x14" sneaks in
print(re.findall(r"\d+.\d+", "Version 3x14 and 2.718"))   # ['3x14', '2.718']

# Escaped: \. means "a real period"
print(re.findall(r"\d+\.\d+", "Version 3x14 and 2.718"))  # ['2.718']

# Prices need an escaped $ (otherwise $ means "end of string")
print(re.findall(r"\$\d+\.\d{2}", "Total: $19.99 (was $25.00)"))  # ['$19.99', '$25.00']

That's why the FASTA pattern in 22.1 writes \| for each pipe. When the literal text comes from somewhere else (say, a user's search box), re.escape() adds the backslashes for you: print(re.escape("C++")) shows C\+\+.

Connection to Chapter 7: Remember str.isdigit(), str.isalpha(), and str.isalnum() from Chapter 7? They test whether an entire string consists of certain character types. Character classes in regex let you match those same categories within a larger pattern.

22.4 Quantifiers: +, *, ?, {n,m}

A character class matches one character. Quantifiers specify how many times a character or class should repeat.

Let's put these together with character classes to build useful patterns.

import re

text = "Order #12345 shipped on 2025-03-14. Contact: [email protected]"

# Find the order number (# followed by digits)
order = re.search(r"#(\d+)", text)
print(order.group())   # #12345
print(order.group(1))  # 12345  (just the digits — we'll explain groups in 22.6)

# Find the date (four digits, dash, two digits, dash, two digits)
date = re.search(r"\d{4}-\d{2}-\d{2}", text)
print(date.group())    # 2025-03-14

Elena's Phone Number Problem

Elena Vasquez — our nonprofit data analyst — receives donor contact information in wildly inconsistent formats. Phone numbers appear as:

(555) 867-5309
555-867-5309
555.867.5309
5558675309
555 867 5309

With quantifiers and character classes, she can write one pattern that handles all of them:

# Optional opening paren, 3 digits, optional closing paren,
# optional separator (space, dash, or dot), 3 digits,
# optional separator, 4 digits
phone_pattern = r"\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}"

phones = [
    "(555) 867-5309",
    "555-867-5309",
    "555.867.5309",
    "5558675309",
    "555 867 5309",
]

for phone in phones:
    match = re.search(phone_pattern, phone)
    if match:
        print(f"  Matched: {match.group()}")
# All five lines print "Matched: ..." with the full number

Let's break down that pattern piece by piece:

This is the power of regex. One pattern, five formats. (It's also a little too forgiving — it would accept a lopsided (555 867-5309 because each parenthesis is optional on its own. Every pattern is a trade-off between strictness and simplicity; for cleaning donor records, this one is fine.)

Check Your Understanding #2

What does the pattern r"[A-Z][a-z]*" match? Give three example strings that would match.

22.5 Anchors and Boundaries: ^, $, \b

So far, our patterns float — they match wherever they appear in the string. Anchors pin a pattern to a specific position.

Start and End Anchors

import re

lines = [
    "ERROR: disk full",
    "WARNING: disk space low",
    "INFO: disk check passed",
    "Disk ERROR found",
]

# Only lines that START with ERROR
for line in lines:
    if re.search(r"^ERROR", line):
        print(line)
# Output: ERROR: disk full

# Only lines that END with a specific word
for line in lines:
    if re.search(r"passed$", line):
        print(line)
# Output: INFO: disk check passed

One quirk worth knowing: $` also matches just *before* a newline at the very end of the string, so `re.search(r"^\d+$", "123\n") succeeds. That's handy for lines read from a file, but when you're validating input and want "this exact string and nothing else," re.fullmatch() is the stricter choice — re.fullmatch(r"\d+", "123\n") returns None.

Word Boundaries

The \b anchor is incredibly useful. It matches the boundary between a word character and a non-word character — it doesn't consume any characters, it just asserts a position.

text = "cat concatenate caterpillar scat category"

# Without \b — finds "cat" inside other words too
print(re.findall(r"cat", text))
# ['cat', 'cat', 'cat', 'cat', 'cat']

# With \b — finds only the standalone word "cat"
print(re.findall(r"\bcat\b", text))
# ['cat']

# Words that START with "cat"
print(re.findall(r"\bcat\w*", text))
# ['cat', 'caterpillar', 'category']

Notice that concatenate and scat are missing from the last result. The \b demands a boundary right before the c, and in both words the c is preceded by another letter.

Pitfall: Remember that \b needs a raw string! Without the r prefix, Python interprets \b as a backspace character, and your pattern silently fails to match what you expect.

Dr. Patel's FASTA Processing with Anchors

Dr. Patel's DNA FASTA files mix header lines (starting with >) and sequence lines (containing only nucleotide characters). Anchors make it easy to separate them — and to catch lines that are neither:

fasta_data = """>gi|12345|ref|NM_001.2| BRCA1 gene
ATCGATCGATCGATCGATCG
GCTAGCTAGCTAGCTAGCTA
>gi|67890|ref|NM_002.1| TP53 gene
TTTTAAAACCCCGGGG
TTAGXXCC"""

lines = fasta_data.strip().split("\n")

headers = [line for line in lines if re.search(r"^>", line)]
sequences = [line for line in lines if re.search(r"^[ACGT]+$", line)]

print(f"Headers:   {len(headers)}")    # Headers:   2
print(f"Sequences: {len(sequences)}")  # Sequences: 3

The pattern ^[ACGT]+$` means "the entire line consists of one or more nucleotide characters and nothing else." That's precise: the corrupted line `TTAGXXCC` lands in neither list, because `^` and `$ force every character between them to be an A, C, G, or T. Without the anchors, [ACGT]+ would happily find TTAG inside it.

(For the header check alone, line.startswith(">") would be just as good — see Section 22.9. The anchors earn their keep on the sequence check.)

22.6 Groups and Capturing: (pattern) and Named Groups

Groups are where regex goes from "find this pattern" to "extract data." Parentheses create a capture group — they tell the regex engine "I want to remember this part separately."

Basic Capture Groups

import re

log = "2025-03-14 08:23:17 ERROR Database connection timeout"

# Capture year, month, day separately
match = re.search(r"(\d{4})-(\d{2})-(\d{2})", log)
if match:
    print(match.group())   # 2025-03-14  (full match)
    print(match.group(1))  # 2025        (first group)
    print(match.group(2))  # 03          (second group)
    print(match.group(3))  # 14          (third group)
    print(match.groups())  # ('2025', '03', '14')

Groups are numbered left to right, starting at 1. group(0) (or just group()) is the entire match.

findall() with Groups

When your pattern contains groups, findall() returns the group contents — not the full match:

text = "Born: 1990-05-21, Graduated: 2012-06-15, Hired: 2015-09-01"

# Without groups — returns full matches
dates = re.findall(r"\d{4}-\d{2}-\d{2}", text)
print(dates)  # ['1990-05-21', '2012-06-15', '2015-09-01']

# With groups — returns tuples of group contents
parts = re.findall(r"(\d{4})-(\d{2})-(\d{2})", text)
print(parts)  # [('1990', '05', '21'), ('2012', '06', '15'), ('2015', '09', '01')]

With exactly one group, you get a plain list of strings instead of tuples: re.findall(r"(\d{4})-\d{2}-\d{2}", text) returns ['1990', '2012', '2015']. This makes findall() with groups a powerful data extraction tool — and a classic source of surprise when you add parentheses for some other reason and your full matches suddenly vanish.

Named Groups

Group numbers are fragile — if you add a group, all the numbers shift. Named groups let you refer to groups by name:

pattern = r"(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})"
match = re.search(pattern, "Event on 2025-03-14")
if match:
    print(match.group("year"))   # 2025
    print(match.group("month"))  # 03
    print(match.group("day"))    # 14
    print(match.groupdict())     # {'year': '2025', 'month': '03', 'day': '14'}

The syntax (?P<name>pattern) creates a named group. The .groupdict() method returns all named groups as a dictionary — perfect for converting matched data into structured form.

Elena's Email Extraction

Elena needs to pull email addresses out of messy donor records. Here's a pattern that captures the username and domain separately:

email_pattern = r"(?P<user>[\w.+-]+)@(?P<domain>[\w.-]+\.\w{2,})"

records = [
    "Jane Doe <[email protected]> donated $500",
    "Contact: [email protected]",
    "Reply to [email protected] for details",
]

for record in records:
    match = re.search(email_pattern, record)
    if match:
        print(f"  User: {match.group('user'):20s}  "
              f"Domain: {match.group('domain')}")

Output:

  User: jane.doe              Domain: example.com
  User: bob_smith             Domain: harbor-nonprofit.org
  User: info                  Domain: example.co.uk

Non-Capturing Groups

Sometimes you need parentheses for grouping (to apply a quantifier to a multi-character sequence) but you don't want to capture the contents. Use (?:pattern):

laughs = "hahaha haha"

# The group makes + repeat "ha" as a unit, but findall() returns the
# group, and a repeated group remembers only its LAST repetition
print(re.findall(r"(ha)+", laughs))    # ['ha', 'ha']

# Non-capturing group: same grouping, nothing stored, full matches back
print(re.findall(r"(?:ha)+", laughs))  # ['hahaha', 'haha']

Without any parentheses, ha+ would mean "an h, then one or more as" — it matches haaa, not hahaha. The quantifier applies only to the single item right before it, and a group is how you make "the item" longer than one character.

Alternation: This or That

The vertical bar | means "or": cat|dog matches either word. It has the lowest priority of anything in a pattern, so it splits the whole pattern in two unless a group fences it in:

print(re.findall(r"cat|dog", "hotdog catalog"))   # ['dog', 'cat']

# Without a group, | splits everything: "gra" OR "ey"
print(re.findall(r"gra|ey", "gray grey"))          # ['gra', 'ey']

# With a non-capturing group, only the middle letter varies
print(re.findall(r"gr(?:a|e)y", "gray grey"))      # ['gray', 'grey']

You'll see alternation in log parsing (INFO|WARNING|ERROR) and in TaskFlow's search (meeting|call) later in this chapter.

22.7 Substitution with re.sub()

re.sub(pattern, replacement, string) finds all matches and replaces them. It's str.replace() on steroids.

Simple Replacement

import re

text = "Call 555-1234 or 555-5678 for info."

# Redact phone numbers
redacted = re.sub(r"\d{3}-\d{4}", "XXX-XXXX", text)
print(redacted)  # Call XXX-XXXX or XXX-XXXX for info.

Using Groups in Replacements

You can reference captured groups in the replacement string using \1, \2, etc. (or \g<name> for named groups):

# Reformat dates from YYYY-MM-DD to MM/DD/YYYY
text = "Start: 2025-03-14, End: 2025-06-30"
reformatted = re.sub(
    r"(\d{4})-(\d{2})-(\d{2})",
    r"\2/\3/\1",
    text,
)
print(reformatted)  # Start: 03/14/2025, End: 06/30/2025

The replacement \2/\3/\1 means: second group (month), slash, third group (day), slash, first group (year).

Using a Function as Replacement

For complex replacements, pass a function instead of a string. The function receives the match object and returns the replacement string:

def censor_email(match):
    """Replace the middle of an email username with asterisks."""
    user = match.group("user")
    domain = match.group("domain")
    if len(user) > 2:
        censored = user[0] + "*" * (len(user) - 2) + user[-1]
    else:
        censored = user[0] + "*"
    return f"{censored}@{domain}"

text = "Contact [email protected] or [email protected]"
result = re.sub(
    r"(?P<user>[\w.+-]+)@(?P<domain>[\w.-]+\.\w{2,})",
    censor_email,
    text,
)
print(result)  # Contact j******[email protected] or b*[email protected]

re.sub() with a Count Limit

The optional count parameter limits how many replacements to make:

text = "aaa bbb ccc ddd"
print(re.sub(r"\w+", "X", text))           # X X X X
print(re.sub(r"\w+", "X", text, count=2))  # X X ccc ddd

22.8 Greedy vs. Lazy Matching

This is where beginners get bitten. By default, quantifiers are greedy — they match as much text as possible.

import re

html = '<b>bold</b> and <i>italic</i>'

# Greedy: .* matches as MUCH as possible
match = re.search(r"<.*>", html)
print(match.group())
# <b>bold</b> and <i>italic</i>     <-- matched too much!

The .* ate everything from the first < to the last > — which isn't what you wanted. Here's how it got there: after the opening <, the greedy .* grabs the entire rest of the string. Now the pattern still needs a >, and there's nothing left, so the engine backtracks — gives back one character at a time — until a > is available. The first > it reaches while backing up from the end is the last one in the string.

Lazy Quantifiers

Adding ? after a quantifier makes it lazy (also called non-greedy or reluctant). Lazy quantifiers match as little as possible.

html = '<b>bold</b> and <i>italic</i>'

# Lazy: .*? matches as LITTLE as possible
tags = re.findall(r"<.*?>", html)
print(tags)  # ['<b>', '</b>', '<i>', '</i>']

# Extract tag contents
contents = re.findall(r"<\w+>(.*?)</\w+>", html)
print(contents)  # ['bold', 'italic']

The .*? works the other way around: it starts by matching nothing, and after each character it checks whether the rest of the pattern (in <.*?>, just the >) can match yet. So it stops at the first > it finds, not the last.

When to Use Each

• Greedy (default): when you want the longest possible match. Usually right for patterns that don't have an ending delimiter, like \d+ (match all consecutive digits).
• Lazy: when you're matching content between delimiters, like extracting text between HTML tags, quotes, or brackets.

Debugging Tip: If your regex matches more than you expected, check whether a greedy quantifier is gobbling up too much. Adding ? to make it lazy is usually the fix. An even more explicit fix is a negated class that says what can't appear: <[^>]*> means "a <, then anything except >, then >" — it can't overshoot, greedy or not.

Check Your Understanding #3

What does re.findall(r'"(.*?)"', 'She said "hello" and "goodbye"') return?

22.9 When NOT to Use Regex

Here's an important truth that regex enthusiasts often forget: string methods are frequently better than regex.

Prefer String Methods When...

1. You're looking for a fixed string:

# YES — simple and clear
if "ERROR" in log_line:
    ...

# OVERKILL — regex for a fixed string
if re.search(r"ERROR", log_line):
    ...

2. You're splitting on a fixed delimiter:

# YES
parts = line.split(",")

# OVERKILL
parts = re.split(r",", line)

3. You're doing a simple replacement:

# YES
clean = text.replace("  ", " ")

# OVERKILL
clean = re.sub(r"  ", " ", text)

4. You're checking start/end:

# YES
if filename.endswith(".csv"):
    ...

# OVERKILL
if re.search(r"\.csv$", filename):
    ...

Use Regex When...

• The pattern involves variability: multiple formats, optional parts, unknown length
• You need to extract parts of the match (capture groups)
• You're matching character classes (any digit, any letter)
• The delimiter is complex or variable: split on any whitespace, any punctuation
• You need anchored matching: word boundaries, start/end of line

One more boundary: when the text is a structured format with its own rules — CSV, JSON, HTML — use a real parser rather than either tool. A quoted CSV field can contain commas, and HTML tags nest, so <.*?> (Section 22.8) is fine for a demo but breaks on real web pages. The csv and json modules from Chapter 21 and Beautiful Soup in Chapter 24 already know those rules; regex is for the messy, unstructured text in between.

The Readability Test

If a colleague can't understand your regex pattern in 30 seconds, consider whether string methods would be clearer — even if they take more lines. Code is read far more often than it's written.

# This regex is clever but opaque:
re.sub(r"(\w)(\w+)", lambda m: m.group(1).upper() + m.group(2).lower(), text)

# This is longer but immediately clear:
words = text.split()
result = " ".join(word.capitalize() for word in words)

For an ordinary sentence like "the quick brown fox", both produce "The Quick Brown Fox". (They disagree on edge cases — the regex version leaves one-letter words like "a" alone and capitalizes after hyphens; the split() version collapses runs of spaces — which is itself a hint that the clever version is harder to reason about.) The second version is better code.

String Methods vs. Regex: Quick Decision Guide

Theme: Errors are information, not failures (Ch 3). Don't beat yourself up when a regex doesn't work on the first try. Regex development is inherently iterative — write a simple pattern, test it, refine it, test again. Tools like regex101.com let you test patterns interactively and see exactly what matches.

22.10 Quick Reference: Regex Syntax at a Glance

Keep this table bookmarked. You'll refer to it constantly when writing regex.

Characters and Classes

Quantifiers

Anchors and Boundaries

Groups and Alternation

Common re Functions

22.11 Compiling Patterns with re.compile()

If you use the same pattern repeatedly — say, inside a loop that processes thousands of lines — you can compile it once for better performance:

import re

# Compile the pattern once
date_pattern = re.compile(r"(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})")

log_lines = [
    "2025-03-14 08:23:17 ERROR disk full",
    "2025-03-14 08:24:01 INFO recovery started",
    "2025-03-14 08:25:33 WARNING disk space low",
]

for line in log_lines:
    match = date_pattern.search(line)  # Use the compiled pattern object
    if match:
        print(f"  {match.group('year')}/{match.group('month')}/{match.group('day')}")

The compiled pattern object has the same methods (search, match, findall, sub, split) as the re module functions, but you call them on the pattern object instead.

Performance note: Python caches the compiled versions of recently used patterns, even when you call re.search() directly, so compiling yourself only matters when you have many distinct patterns or are processing very large datasets in tight loops. For typical scripts, the performance difference is negligible. Compile when it makes the code cleaner (giving the pattern a descriptive name), not just for speed.

22.12 Flags: Modifying Regex Behavior

Regex flags change how the pattern engine operates. The most useful ones:

import re

# Case-insensitive search
text = "Error: file not found. ERROR: disk full. error: timeout."
errors = re.findall(r"error", text, re.IGNORECASE)
print(errors)  # ['Error', 'ERROR', 'error']

# Multiline: ^ matches start of each line
log = """INFO: started
ERROR: disk full
WARNING: low memory
ERROR: timeout"""

error_lines = re.findall(r"^ERROR:.*", log, re.MULTILINE)
print(error_lines)  # ['ERROR: disk full', 'ERROR: timeout']

Combine flags with the | operator: re.IGNORECASE | re.MULTILINE.

Pitfall: With re.sub() and re.split(), always pass flags by name. Their fourth positional argument is count (or maxsplit), not flags, and re.IGNORECASE is secretly just the number 2. So re.sub(r"error", "X", text, re.IGNORECASE) quietly means "replace at most 2 matches, case-sensitively." Write re.sub(r"error", "X", text, flags=re.IGNORECASE). Python 3.13 warns about the positional form (a DeprecationWarning) and plans to make these arguments keyword-only.

22.13 Project Checkpoint: TaskFlow v2.1

Time to put regex to work in our progressive project. In Chapter 21, TaskFlow v2.0 added CSV/JSON import/export and an inspirational-quote API. Now we'll add two regex-powered features:

1. Advanced search — search tasks with regex patterns, not just plain keywords
2. Natural language date parsing — interpret inputs like "next Tuesday" or "tomorrow" as actual dates

Like v2.0, the full listing in code/project-checkpoint.py is a stand-alone, dictionary-based script whose task dictionaries have the same keys v1.5's Task.to_dict() writes; a due date goes in the due_date field as a "YYYY-MM-DD" string, or "" for none, exactly as on a Task. It leaves out the export/import and quote code so the new regex features are easy to see.

Feature 1: Regex-Powered Search

The current search_tasks() from v2.0 uses keyword in title.lower() — a simple substring match. Let's upgrade it to accept regex patterns:

import re

def search_tasks_regex(tasks: list[dict], pattern: str) -> list[dict]:
    """Search tasks using a regex pattern.

    Falls back to plain substring search if the pattern is
    not valid regex (e.g., "(" or "[abc"). Note that "C++" IS valid
    since Python 3.11 (++ is a possessive quantifier), so it matches
    any title containing a "c"; wrap input in re.escape() if you want
    purely literal searches.
    """
    try:
        compiled = re.compile(pattern, re.IGNORECASE)
    except re.error:
        # Invalid regex — treat as literal substring
        print("  (Invalid regex — using plain text search)")
        return [t for t in tasks if pattern.lower() in t["title"].lower()]

    return [t for t in tasks if compiled.search(t["title"])]

Now users can search with patterns like: - "^Buy" — tasks that start with "Buy" - "meeting|call" — tasks containing "meeting" or "call" - "\d{4}" — tasks containing a 4-digit number - "report$" — tasks ending with "report"

The try/except re.error is the EAFP style from Chapter 11: a pattern like ( or [abc can't be compiled, so we quietly fall back to a plain search. (Python 3.13 gave this exception the clearer name re.PatternError; re.error still works as an alias.) Be aware that some strings that look invalid aren't. Since Python 3.11, ++ is a "possessive quantifier," so C++ is a legal pattern meaning "one or more C's" — and it will match every task with a "c" in it. If your users mostly want literal searches, re.escape(pattern) is the safer default.

Feature 2: Natural Language Date Parsing

This is a simplified date parser — it handles common phrases, not full natural language. The key insight: each phrase is a pattern that maps to a date calculation.

import re
from datetime import datetime, timedelta

WEEKDAYS = {
    "monday": 0, "tuesday": 1, "wednesday": 2, "thursday": 3,
    "friday": 4, "saturday": 5, "sunday": 6,
}

def parse_natural_date(text: str) -> datetime | None:
    """Parse simple natural language date expressions.

    Supports: 'today', 'tomorrow', 'next <weekday>' (the nearest
    upcoming one, never today), 'in N days', and 'YYYY-MM-DD'.
    """
    text = text.strip().lower()
    today = datetime.now().replace(hour=0, minute=0, second=0, microsecond=0)

    # "today"
    if re.fullmatch(r"today", text):
        return today

    # "tomorrow"
    if re.fullmatch(r"tomorrow", text):
        return today + timedelta(days=1)

    # "next <weekday>"
    match = re.fullmatch(r"next\s+(\w+)", text)
    if match:
        day_name = match.group(1)
        if day_name in WEEKDAYS:
            target = WEEKDAYS[day_name]
            current = today.weekday()
            days_ahead = (target - current) % 7
            if days_ahead == 0:
                days_ahead = 7  # "next Monday" on a Monday means 7 days
            return today + timedelta(days=days_ahead)

    # "in N days"
    match = re.fullmatch(r"in\s+(\d+)\s+days?", text)
    if match:
        n = int(match.group(1))
        return today + timedelta(days=n)

    # ISO format: YYYY-MM-DD
    match = re.fullmatch(r"(\d{4})-(\d{2})-(\d{2})", text)
    if match:
        try:
            return datetime(
                int(match.group(1)),
                int(match.group(2)),
                int(match.group(3)),
            )
        except ValueError:
            return None

    return None

This function uses re.fullmatch() to ensure the entire input matches a known pattern — not just a substring. Each branch handles one format, extracts the relevant data with groups, and computes the result.

Two details are worth a second look. First, the regex only checks the shape of an ISO date; "2025-02-30" has the right shape, so it's datetime() that rejects it with a ValueError. Regex for format, real code for meaning. Second, "next Friday" is ambiguous in everyday English. This version picks the nearest upcoming Friday — on a Wednesday that's two days away — and only jumps a full week when today is Friday. Whatever rule you choose, write it down in the docstring so users aren't surprised.

Integrating Into the Main Loop

Here's add_task() from the full program, with the get_valid_priority() helper it shares with v2.0. (save_tasks() and DATA_FILE are unchanged from v2.0.)

def get_valid_priority() -> str:
    """Prompt until the user types high, medium, or low (as in v1.0)."""
    while True:
        priority = input("  Priority (high/medium/low): ").strip().lower()
        if priority in PRIORITY_ORDER:
            return priority
        print("  Please type high, medium, or low.")


def add_task(tasks: list[dict]) -> None:
    """Add a new task with title, priority, category, and optional due date."""
    title = input("  Task title: ").strip()
    if not title:
        print("  Title cannot be empty.")
        return

    priority = get_valid_priority()

    category = input("  Category [general]: ").strip() or "general"

    # NEW: natural language due date
    due_input = input(
        "  Due date (e.g., 'tomorrow', 'next Friday', "
        "'in 3 days', '2025-04-01') [none]: "
    ).strip()

    due_date = None
    if due_input:
        due_date = parse_natural_date(due_input)
        if due_date is None:
            print(f"  Could not parse '{due_input}' — no due date set.")
        else:
            print(f"  Due date set: {due_date.strftime('%Y-%m-%d')}")

    task = {
        "title": title,
        "priority": priority,
        "category": category,
        "due_date": due_date.strftime("%Y-%m-%d") if due_date else "",
        "completed": False,
        "created_at": datetime.now().isoformat(timespec="minutes"),
    }
    tasks.append(task)
    save_tasks(tasks, DATA_FILE)
    print(f"  Added: '{title}' [{priority}, {category}]")

What we built: TaskFlow v2.1 adds regex search (users can filter tasks with patterns like "^Buy" or "meeting|call") and natural language date parsing (users type "next Tuesday" and get a real date). Both features are powered by the re module.

Chapter Summary

Regular expressions give you a concise, powerful language for describing text patterns. You learned to:

1. Import and use the re module: search, match, fullmatch, findall, finditer, sub, split
2. Build patterns with character classes ([abc], \d, \w, \s), quantifiers (+, *, ?, {n,m}), anchors (^, $`, `\b`), and escaped metacharacters (`\.`, `\$)
3. Extract data with capture groups (pattern) and named groups (?P<name>pattern), and choose between alternatives with |
4. Replace text with re.sub(), including group references and function-based replacements
5. Control greediness by adding ? after quantifiers for lazy matching
6. Know the limits — string methods are often simpler and clearer for straightforward tasks

Regex is a skill that rewards practice. You won't memorize every metacharacter on first reading — nobody does. Instead, keep the quick reference table handy, use tools like regex101.com to test patterns interactively, and build your fluency one pattern at a time.

Spaced Review: You'll see regex again in Chapter 24 (web scraping — checking and cleaning the text you pull out of web pages) and the Capstone 3 (data dashboard — parsing mixed-format datasets). The pattern-matching mindset from this chapter will also help you think about data validation wherever it appears.

What's next: In Chapter 23, we shift from what you can do with Python to how you manage Python itself — virtual environments, pip, requirements.txt, and the ecosystem of third-party libraries that make Python the language of choice for everything from web development to data science.