Chapter 35 Key Takeaways: Natural Language Processing for Business Text

The Core Idea

Business generates enormous amounts of text — support tickets, reviews, surveys, emails, contracts. Classical NLP gives you tools to process that text systematically, surfacing patterns that would take weeks of manual reading to find. The goal is not perfect analysis of individual sentences; it is reliable pattern detection across hundreds or thousands of documents.

What You Learned

Text Preprocessing Is the Foundation

Before any analysis, raw text must be cleaned and normalized. The standard pipeline — lowercase, remove punctuation, tokenize, remove stopwords, lemmatize — transforms noisy human language into analyzable tokens. The right combination of steps depends on your task: keep stopwords for sentiment analysis (you need "not"), remove them for keyword extraction.

Key tools: re for regex cleaning, NLTK for tokenization and stopword removal, WordNetLemmatizer for linguistically accurate base forms.

TextBlob Provides Fast, Accessible Sentiment

For most business text analysis tasks, TextBlob's polarity score (−1.0 to +1.0) is sufficient. It is a lexicon-based approach — it looks up words in a pre-built sentiment dictionary — which makes it fast and transparent. Use it for aggregate analysis: average polarity by category, percentage of negative responses, trend over time. Do not use individual scores as authoritative verdicts.

Rule of thumb: TextBlob accuracy is typically 70-80% on business text against human labels. This is good enough for finding patterns in hundreds of tickets; it is not good enough for automated decisions that affect individual customers.

TF-IDF Finds What Makes Text Distinctive

Raw word frequency tells you what words appear most often. TF-IDF tells you what words are most distinctive within specific documents or categories. Use TfidfVectorizer with ngram_range=(1, 2) to capture both single words and two-word phrases. The bigrams are often more informative: "shipping delay" is more useful than "shipping" or "delay" alone.

spaCy's NER Is Production-Ready

Named entity recognition with spaCy's en_core_web_sm model extracts people, organizations, dates, monetary amounts, and locations from text with high accuracy on formal business documents. Use nlp.pipe() for batch processing — it is significantly faster than calling nlp() in a loop.

Business applications: Extract all company names from competitor mentions, pull all dates from contracts, identify all monetary amounts in invoices.

Text Classification Can Be Rule-Based or ML-Based

Keyword rules are transparent, auditable, and require no labeled data. They work well when categories are well-defined and domain-specific phrases are reliable signals. Machine learning (TF-IDF + Naive Bayes via scikit-learn Pipeline) outperforms keyword rules when you have labeled training data (200+ labeled examples is a practical minimum). Both approaches belong in your toolkit.

LDA Discovers Themes Without Labels

Latent Dirichlet Allocation finds latent topics in a collection of documents without predefined categories. It is particularly useful when you do not know what the themes are before you start. Specify the number of topics, check coherence scores to validate your choice, and interpret the topics yourself — LDA returns word lists, not labels.

NLP Has Real, Specific Limits

• Sarcasm: Lexicon-based tools consistently misclassify sarcastic text as positive.
• Complex negation: "Not entirely dissatisfied" is hard to score correctly.
• Short text: Single-sentence reviews have high variance in polarity scores.
• Domain-specific language: A "positive" result in medicine is not good news.
• Changing language: Models trained on older data may miss emerging terminology.

Design your systems to work with these limits: use aggregate statistics, validate on your specific domain, keep humans in the loop for edge cases.

Business Impact Patterns

Volume processing: The primary value of NLP for most businesses is converting unread text into structured data. A Python script running TextBlob on 4,200 tickets takes a few minutes; reading them would take weeks.

Triage: Combining polarity with ticket age produces an urgency score that helps support teams address the most critical cases first.

Operational insights from language patterns: When shipping damage tickets mention "third party carrier" in 20% of cases, or Monday is disproportionately represented in shipping delay tickets, those patterns are operationally actionable and would not be visible without processing the text.

What clients or customers are not saying: The absence of certain topics is as informative as their presence. When fewer than 10% of survey responses mention pricing, that is a signal about perceived value — or about what clients consider worth commenting on.

Practical Checklist

Before starting any business NLP project:

• [ ] Define the question you are trying to answer before writing code
• [ ] Check whether you have labeled data (opens ML classification options) or not (rules or unsupervised methods)
• [ ] Understand your text: formal (contracts) vs. informal (reviews)? Long (reports) vs. short (tweets)?
• [ ] Choose preprocessing steps appropriate to your task (sentiment = keep stopwords; topics = remove them)
• [ ] Plan how results will be used: aggregate reporting or individual decisions?
• [ ] Validate on a sample before applying to the full dataset

Library Reference

One-Sentence Summary

Natural Language Processing lets you ask structured questions of unstructured text — and classical Python NLP tools make that practical for any business analyst willing to spend an afternoon learning them.