Chapter 10: Working with Text Data — String Methods, Regular Expressions, and Extracting Meaning

         manufacturer        code
0     Pfizer-BioNTech     BNT162b2
1             Moderna    mRNA-1273
2         AstraZeneca      AZD1222
3                 NaN          NaN
4             Sinovac    CoronaVac

Row 3 (Johnson & Johnson) didn't match because it has no code in parentheses. That's fine — NaN tells us which entries need different handling.

Let's break down that pattern piece by piece: - ^ — start of string - (?P<manufacturer>[A-Za-z&\s-]+?) — capture group named "manufacturer": one or more letters, ampersands, spaces, or hyphens (lazy match) - \s* — zero or more spaces - \( — a literal opening parenthesis (escaped because ( is special in regex) - (?P<code>[^)]+) — capture group named "code": one or more characters that aren't a closing parenthesis - \) — a literal closing parenthesis

Check Your Understanding

1. What's the difference between re.search(r"\d+", text) and re.findall(r"\d+", text)?
2. In the pattern (\d{4})-(\d{2})-(\d{2}), how many capture groups are there?
3. What does str.extract() return when a row doesn't match the pattern?

10.10 Threshold Concept: Regular Expressions as a Mini-Language for Describing Patterns

Threshold Concept Alert: This is a concept that, once you truly grasp it, fundamentally changes how you see text data. It may feel uncomfortable at first.

Here's the mental shift: a regular expression is not a string. It's a program.

When you write r"\d{4}-\d{2}-\d{2}", you're not writing a string that somehow matches dates. You're writing instructions in a mini programming language. Those instructions say:

1. Match a digit. Do this four times.
2. Match a literal hyphen.
3. Match a digit. Do this two times.
4. Match a literal hyphen.
5. Match a digit. Do this two times.

This language has its own vocabulary (\d, \w, \s), its own control structures (quantifiers, alternation), its own grouping mechanism (parentheses), and its own anchoring system (^, $). It's a language within a language.

Why does this matter? Because once you see regex as a language for describing patterns, three things change:

First, you stop trying to memorize patterns and start composing them. You don't memorize "the regex for a phone number." Instead, you think: "A phone number is three digits, a separator, three digits, a separator, four digits" — and you compose the pattern: \d{3}[-.\s]\d{3}[-.\s]\d{4}.

Second, you start seeing text as having structure. That messy free-text field? It's not chaos. It has patterns. Dates follow patterns. Product codes follow patterns. Addresses follow patterns. Regex is the tool for describing those patterns to a computer.

Third, you understand why regex can be both powerful and dangerous. A programming language that lets you express complex ideas in a few characters is powerful. But dense code is hard to read, debug, and maintain. A 50-character regex that nobody can understand is worse than five lines of clear Python that do the same thing.

This is why experienced data scientists follow a rule: use the simplest tool that works. If str.lower() solves your problem, don't use regex. If str.replace("old", "new") works, don't use regex. Save regex for the problems where you genuinely need pattern matching — extracting variable-format data, validating complex formats, finding patterns that can't be described by a literal string.

Before the threshold: "Regex is a weird way to search for text." After the threshold: "Regex is a language for describing the structure of text, and I can compose patterns from simple building blocks."

Threshold Check

1. Explain in your own words why regex is described as a "mini-language" rather than just a search tool.
2. Given a string like "Invoice #2023-0042, Amount: $1,234.56", describe in plain English the structure you see — then sketch a regex pattern for each piece.
3. A colleague wrote the regex ^[A-Z]{2}\d{6}[A-Z]$ and says "it matches passport numbers." Without running it, describe in words what strings this pattern will match.

10.11 Putting It Together: Alternation, Anchors, and Escaping

Let's round out our regex toolkit with three more concepts.

Alternation: Matching One Thing OR Another

The pipe character | means "or":

re.findall(r"cat|dog", "I have a cat and a dog and a catfish")

['cat', 'dog', 'cat']

Use parentheses to limit the scope of alternation:

# Without parentheses: matches "gray" or "grey"
re.findall(r"gray|grey", "The gray cat and grey dog")

# Same thing, more concisely
re.findall(r"gr[ae]y", "The gray cat and grey dog")

In data science, alternation is great for matching multiple variants:

vaccines = pd.Series([
    "Pfizer vaccine",
    "Moderna shot",
    "J&J jab",
    "AstraZeneca injection"
])

# Find any entry mentioning a vaccine (different words)
vaccines.str.contains(
    r"vaccine|shot|jab|injection", case=False
)

0    True
1    True
2    True
3    True
dtype: bool

Anchors: Matching Position, Not Characters

Anchors don't match characters — they match positions in the string.

# Without anchors: matches "cat" anywhere
re.findall(r"cat", "The cat caught a catfish")
# ['cat', 'cat', 'cat']

# With word boundary: matches "cat" as a whole word
re.findall(r"\bcat\b", "The cat caught a catfish")
# ['cat']

Word boundaries (\b) are extremely useful in data science for matching whole words without accidentally matching substrings:

# Searching for the state "IN" (Indiana)
states = pd.Series(["IN", "INDIANA", "MISSING", "IN PROGRESS"])

# Bad: matches "IN" inside other words
states.str.contains("IN")
# All True!

# Better: match "IN" as a complete word
states.str.contains(r"\bIN\b")
# True, False, False, True

Hmm, that last one ("IN PROGRESS") still matched. If you want to match only strings that are exactly "IN":

states.str.match(r"^IN$")
# True, False, False, False

Escaping Special Characters

In regex, characters like ., *, +, ?, (, ), [, ], {, }, ^, $, |, and \ have special meanings. If you want to match them literally, you need to escape them with a backslash:

# The dot matches any character
re.findall(r"3.14", "3.14 and 3X14")
# ['3.14', '3X14']

# Escaped dot matches only a literal dot
re.findall(r"3\.14", "3.14 and 3X14")
# ['3.14']

This comes up constantly with data that contains periods, dollar signs, parentheses, and other punctuation:

# Match a price like "$12.99"
re.findall(r"\$\d+\.\d{2}", "Total: $12.99 plus $3.50 shipping")
# ['$12.99', '$3.50']

The pattern \$\d+\.\d{2} means: a literal dollar sign, one or more digits, a literal dot, exactly two digits.

10.12 Greedy vs. Lazy Matching

This is a subtlety that trips up even experienced regex users.

By default, quantifiers are greedy — they match as much as possible:

text = "<b>bold</b> and <i>italic</i>"
re.findall(r"<.+>", text)

['<b>bold</b> and <i>italic</i>']

The .+ matched everything from the first < to the last >, swallowing all the text in between. That's not what we wanted.

Adding a ? after a quantifier makes it lazy — it matches as little as possible:

re.findall(r"<.+?>", text)

['<b>', '</b>', '<i>', '</i>']

Now .+? matches just enough to reach the nearest >.

In data science, you'll encounter this when extracting text between delimiters:

notes = pd.Series([
    "Diagnosis: (Type 2 Diabetes) Treatment: (Metformin)",
    "Notes: (Patient stable) Follow-up: (2 weeks)"
])

# Greedy: captures everything between first ( and last )
notes.str.extract(r"\((.+)\)")
# 0    Type 2 Diabetes) Treatment: (Metformin
# 1    Patient stable) Follow-up: (2 weeks

# Lazy: captures just the first parenthesized group
notes.str.extract(r"\((.+?)\)")
# 0    Type 2 Diabetes
# 1    Patient stable

The lazy version stops at the first closing parenthesis, which is almost always what you want.

Check Your Understanding

1. What's the difference between re.findall(r"<.+>", html) and re.findall(r"<.+?>", html)?
2. Why does \$ match a literal dollar sign instead of "end of string"?
3. How would you use \b to search for the word "data" without matching "database" or "metadata"?

10.13 Text Normalization: A Systematic Approach

Now that you have both string methods and regex in your toolkit, let's talk about a complete workflow for text normalization — the process of making equivalent text values consistent.

The Normalization Pipeline

Here's the workflow Elena uses in her public health work:

Step 1: Strip whitespace         .str.strip()
Step 2: Standardize case         .str.lower() or .str.upper()
Step 3: Remove punctuation       .str.replace(r"[^\w\s]", "", regex=True)
Step 4: Collapse whitespace      .str.replace(r"\s+", " ", regex=True)
Step 5: Map known variants       .replace(mapping_dict)
Step 6: Verify results           .value_counts()

Let's apply this to a messy column of country names:

df = pd.DataFrame({"country": [
    "  United States  ", "united states", "USA",
    "U.S.A.", "US", " U.S. ", "Brazil", "BRAZIL",
    "  brazil", "United  Kingdom", "U.K.", "UK",
    None, "germany", "Côte d'Ivoire"
]})

# Steps 1-2: Strip and lowercase
df["clean"] = df["country"].str.strip().str.lower()

# Step 3: Remove periods (careful with Côte d'Ivoire)
df["clean"] = df["clean"].str.replace(".", "", regex=False)

# Step 4: Collapse multiple spaces
df["clean"] = df["clean"].str.replace(r"\s+", " ", regex=True)

# Step 5: Map abbreviations to standard names
mapping = {
    "usa": "united states",
    "us": "united states",
    "usa": "united states",
    "uk": "united kingdom",
}
df["clean"] = df["clean"].replace(mapping)

# Step 6: Check results
print(df["clean"].value_counts())

united states     5
brazil            3
united kingdom    3
germany           1
côte d'ivoire     1
Name: clean, dtype: int64

Fourteen messy entries collapsed into five clean values (plus one NaN). That's text normalization at work.

When Mapping Isn't Enough: Fuzzy Matching

Sometimes the variations are too numerous or unpredictable to map manually. Consider medication names:

meds = pd.Series([
    "metformin", "Metformin", "metforman",
    "metphormin", "METFORMIN HCL", "metformin 500mg"
])

The third and fourth entries are misspellings. A simple mapping dictionary can't catch every possible misspelling. For these cases, there are fuzzy matching libraries like fuzzywuzzy or thefuzz, which we'll mention in the Further Reading. For now, know that this problem exists, and that regex combined with string methods can handle most standardization tasks — but not misspellings.

10.14 Tokenization: Breaking Text into Words

Tokenization is the process of splitting text into individual words (or "tokens"). It's the first step in any text analysis.

The simplest approach uses .str.split():

responses = pd.Series([
    "The vaccine was safe and effective",
    "I experienced mild side effects",
    "No issues at all"
])

# Split into word lists
responses.str.split()

0    [The, vaccine, was, safe, and, effective]
1     [I, experienced, mild, side, effects]
2                        [No, issues, at, all]
dtype: object

For counting words, combine with .str.len():

responses.str.split().str.len()

0    6
1    5
2    4
dtype: int64

For more sophisticated tokenization (handling punctuation, contractions, and special cases), libraries like NLTK or spaCy are the professional tools. But for the kind of text wrangling you'll do in data cleaning, str.split() combined with regex is usually sufficient.

Counting Specific Words

Want to know how many times a specific word appears in each entry?

reviews = pd.Series([
    "Great product, really great quality",
    "Good but not great",
    "Great great great!"
])

reviews.str.lower().str.count(r"\bgreat\b")

0    2
1    1
2    3
dtype: int64

The \b word boundaries ensure we count "great" as a whole word, not as part of "greatest" or "greatly."

10.15 When to Use Regex vs. String Methods: A Decision Guide

This might be the most important section in the chapter. Regex is powerful, but it's also easy to overuse. Here's a decision guide:

Use simple string methods when: - You're doing case conversion (.str.lower(), .str.upper()) - You're stripping whitespace (.str.strip()) - You're replacing a known, fixed substring (.str.replace("old", "new", regex=False)) - You're splitting on a simple delimiter (.str.split(",")) - You're checking for a known, fixed substring (.str.contains("exact text"))

Use regex when: - You need to match a pattern rather than a fixed string ("any sequence of digits") - You need to extract part of a string (capture groups with .str.extract()) - You need to match with flexibility (one word OR another, optional characters) - You need anchoring (must start with, must end with, whole word match) - You need to match repetition (exactly 3 digits, one or more letters)

Avoid regex when: - A string method solves the problem just as well (simpler is better) - The pattern would be unreadable (more than ~30 characters — consider breaking it up) - You're trying to parse a structured format like HTML or JSON (use a proper parser) - You're trying to match natural language meaning, not structure (use NLP tools)

Here's a concrete comparison:

# Task: Check if a string contains "covid"

# String method — simple, clear, fast
df["text"].str.contains("covid", case=False)

# Regex — unnecessary for this task
df["text"].str.contains(r"(?i)covid")

# Use the string method. It's clearer.

# Task: Extract a 5-digit ZIP code from an address

# String method — awkward, fragile
# (How do you know which 5 digits are the ZIP?)

# Regex — this is what it's for
df["address"].str.extract(r"(\d{5})(?:-\d{4})?$")

The guiding principle: start simple, escalate to regex only when you need pattern matching. Your future self (and your colleagues) will thank you.

10.16 Project Checkpoint: Extracting Vaccine Manufacturers from Messy Text

Let's apply everything we've learned to Elena's vaccination project. Her dataset has a column called vaccine_info that contains free-text descriptions of vaccines. She needs to extract two things:

1. The manufacturer name
2. Whether it's a primary dose or a booster

Here's a representative sample:

vaccine_data = pd.DataFrame({
    "record_id": range(1, 11),
    "vaccine_info": [
        "Pfizer-BioNTech COVID-19 Vaccine (primary series)",
        "MODERNA mRNA vaccine - booster dose",
        "janssen single dose vaccine",
        "Pfizer booster (3rd dose)",
        "AstraZeneca/Oxford vaccine primary",
        "moderna covid vaccine primary dose",
        "PFIZER-BIONTECH (bivalent booster)",
        "J&J/Janssen - single dose",
        "pfizer  primary  series",
        "Unknown vaccine type"
    ]
})

Step 1: Standardize text

vaccine_data["clean"] = (vaccine_data["vaccine_info"]
    .str.strip()
    .str.lower()
    .str.replace(r"\s+", " ", regex=True))

Step 2: Extract manufacturer using pattern matching

# Define manufacturer patterns
manufacturer_patterns = {
    "pfizer": r"pfizer|biontech",
    "moderna": r"moderna|mrna-1273",
    "janssen": r"janssen|j&j|johnson",
    "astrazeneca": r"astrazeneca|oxford|azd1222",
    "sinovac": r"sinovac|coronavac"
}

def identify_manufacturer(text):
    if pd.isna(text):
        return "unknown"
    for name, pattern in manufacturer_patterns.items():
        if re.search(pattern, text):
            return name
    return "unknown"

vaccine_data["manufacturer"] = (vaccine_data["clean"]
    .apply(identify_manufacturer))

Step 3: Identify dose type

vaccine_data["is_booster"] = (vaccine_data["clean"]
    .str.contains(r"booster|3rd dose|bivalent", na=False))

Step 4: Verify results

print(vaccine_data[["vaccine_info", "manufacturer",
                     "is_booster"]])

                                       vaccine_info manufacturer  is_booster
0  Pfizer-BioNTech COVID-19 Vaccine (primary se...       pfizer       False
1       MODERNA mRNA vaccine - booster dose            moderna        True
2             janssen single dose vaccine             janssen       False
3                Pfizer booster (3rd dose)              pfizer        True
4        AstraZeneca/Oxford vaccine primary        astrazeneca       False
5       moderna covid vaccine primary dose             moderna       False
6       PFIZER-BIONTECH (bivalent booster)             pfizer        True
7                 J&J/Janssen - single dose            janssen       False
8                  pfizer  primary  series             pfizer       False
9                    Unknown vaccine type            unknown       False

Ten messy free-text entries, now with clean manufacturer names and booster flags. This is the kind of text wrangling that Elena does daily — and it's the kind of work that makes the difference between a dataset you can analyze and a dataset you can only stare at.

Step 5: Standardize country name variations

While we're at it, let's also tackle the country name standardization that Elena needs:

# Suppose the dataset has these country variations
countries = pd.Series([
    "United States of America", "USA", "U.S.",
    "Republic of Korea", "South Korea", "Korea, South",
    "Viet Nam", "Vietnam",
    "Russian Federation", "Russia",
    "Dem. Rep. Congo", "Democratic Republic of the Congo",
    "DR Congo", "Cote d'Ivoire", "Ivory Coast"
])

# Build a mapping
country_map = {
    "usa": "united states",
    "u.s.": "united states",
    "united states of america": "united states",
    "republic of korea": "south korea",
    "korea, south": "south korea",
    "viet nam": "vietnam",
    "russian federation": "russia",
    "dem. rep. congo": "democratic republic of the congo",
    "dr congo": "democratic republic of the congo",
    "cote d'ivoire": "ivory coast"
}

cleaned = (countries
    .str.strip()
    .str.lower()
    .replace(country_map))

print(cleaned.value_counts())

united states                          3
south korea                            3
democratic republic of the congo       3
vietnam                                2
ivory coast                            2
russia                                 2
Name: count, dtype: int64

Fifteen variations collapsed into six clean country names.

10.17 Common Regex Patterns for Data Science

Here's a reference table of patterns you'll use again and again. You don't need to memorize these — bookmark this page and come back to it.

10.18 The re Module: Beyond findall

We've been using re.search, re.findall, and re.sub. Let's round out our knowledge of the re module with a few more useful features.

re.compile(): Precompiling Patterns

If you're going to use the same pattern many times, compile it first for better performance:

date_pattern = re.compile(r"\d{4}-\d{2}-\d{2}")

# Now use it multiple times
date_pattern.findall("Events on 2023-01-15 and 2023-02-20")
date_pattern.search("Next date: 2023-05-01")

re.IGNORECASE: Case-Insensitive Matching

re.findall(r"pfizer", "Pfizer and PFIZER and pfizer",
           re.IGNORECASE)

['Pfizer', 'PFIZER', 'pfizer']

In pandas, you can achieve this with the case=False parameter or the (?i) flag:

# These are equivalent
df["text"].str.contains("pfizer", case=False)
df["text"].str.contains(r"(?i)pfizer")

re.VERBOSE: Readable Regex with Comments

For complex patterns, the re.VERBOSE flag lets you add whitespace and comments:

phone_pattern = re.compile(r"""
    (\d{3})     # area code
    [-.\s]?     # optional separator
    (\d{3})     # first three digits
    [-.\s]?     # optional separator
    (\d{4})     # last four digits
""", re.VERBOSE)

phone_pattern.findall("Call 555-123-4567 or 800.555.1234")

[('555', '123', '4567'), ('800', '555', '1234')]

This is much more readable than (\d{3})[-.\s]?(\d{3})[-.\s]?(\d{4}). Use re.VERBOSE for any pattern longer than about 20 characters.

10.19 Debugging Regex: When Your Pattern Doesn't Match

Regex debugging is a skill in itself. Here are strategies that work:

Strategy 1: Test with re.findall() on a Small String

Before applying a regex to a column of 50,000 values, test it on a single string:

test = "Pfizer-BioNTech (BNT162b2) 30mcg"
print(re.findall(r"\((\w+)\)", test))
# ['BNT162b2']

Strategy 2: Build Up Incrementally

Don't write the whole pattern at once. Start with the simplest version and add complexity:

text = "Date: 03/15/2023, Amount: $1,234.56"

# Step 1: Match any digits
re.findall(r"\d+", text)
# ['03', '15', '2023', '1', '234', '56']

# Step 2: Match date pattern
re.findall(r"\d{2}/\d{2}/\d{4}", text)
# ['03/15/2023']

# Step 3: Add capture groups
re.findall(r"(\d{2})/(\d{2})/(\d{4})", text)
# [('03', '15', '2023')]

Strategy 3: Use Online Regex Testers

Websites like regex101.com let you type a pattern and a test string, and they show you exactly what matches, which groups are captured, and why. They even explain each part of your pattern in plain English. These tools are invaluable for learning and debugging.

Strategy 4: Check for Common Mistakes

10.20 Real-World Application: Text Data in Public Health

Text data challenges are everywhere in Elena's work:

Patient notes: "Pt reports fever x2 days post-vaccination" needs to be parsed for symptoms ("fever"), duration ("2 days"), and timing ("post-vaccination").

Survey responses: Free-text answers to "Why didn't you get vaccinated?" need to be categorized into themes (cost, access, trust, medical exemption).

Drug names: "Metformin HCl 500mg extended-release tablet" and "metformin hydrochloride 500 mg ER tab" are the same medication but look completely different to a computer.

Address matching: "123 Main St, Apt 4B" and "123 Main Street, #4B" need to be recognized as the same location.

In each case, the workflow is the same: standardize first (case, whitespace, punctuation), then use pattern matching to extract structure, then map variants to canonical forms. The tools you learned in this chapter — .str methods, regex, and capture groups — are the foundation for all of it.

10.21 Spaced Review: Concepts from Chapters 1-9

Learning sticks when you revisit it. Take five minutes to answer these questions from earlier chapters — without looking back. If you can't answer one, that's a signal to review.

From Chapter 1: What are the three types of data science questions? Give an example of each using vaccination data.

From Chapter 3: What's the difference between = and == in Python? Why does this distinction matter?

From Chapter 5: What's the difference between a list and a dictionary? When would you use each?

From Chapter 7: What does df.head() show you? What about df.info()? Which gives you data types?

From Chapter 8: Name three types of missing data (MCAR, MAR, MNAR). Why does it matter which type you have?

From Chapter 9: What's the difference between merge() and concat() in pandas?

Chapter Summary

You started this chapter treating text as an opaque blob of characters. Now you can see the structure inside it.

The .str accessor gives you vectorized string operations — case conversion, stripping, splitting, replacing — that work on entire columns at once, handle missing values gracefully, and make text standardization efficient.

Regular expressions are a mini-language for describing patterns. You learned the building blocks: literal characters, special characters (\d, \w, \s, .), quantifiers (+, *, ?, {n}), character classes ([A-Z], [^0-9]), anchors (^, $, \b), alternation (|), and capture groups (()).

Capture groups combined with .str.extract() let you pull structured data out of unstructured text — extracting dates, codes, names, and numbers from messy free-text fields.

And you learned the most important lesson of all: use the simplest tool that works. String methods first. Regex when you need pattern matching. And always test on a small sample before running against your full dataset.

Text data is everywhere. The skills you built in this chapter will serve you in every data project you ever work on.

What's Next

In Chapter 11, you'll tackle another tricky data type: dates and times. You'll learn to parse date strings, work with time zones, resample time series data, and compute rolling averages — skills that are essential for any analysis involving trends over time. If you're working on the vaccination project, you'll parse date columns and compute rolling 7-day averages of vaccination rates.

This chapter covered text wrangling with pandas .str methods and regular expressions. In the exercises that follow, you'll practice these skills on realistic text data, from cleaning survey responses to extracting information from medical records. Take your time with regex — it rewards patient practice.