Chapter 22 Quiz: Natural Language Processing for Misinformation Detection

Instructions: Attempt each question before revealing the answer. Questions marked (M) are multiple choice; (T/F) are true/false; (SA) are short answer.

Section 1: NLP for Social Good

Question 1 (M): Which of the following tasks is BEST suited to automated NLP-based misinformation detection?

(A) Determining whether a claim about an obscure historical event is factually accurate (B) Matching a new conspiracy theory claim to previously fact-checked versions of the same claim (C) Evaluating whether a satirical article intends to deceive readers (D) Assessing whether an opinion editorial presents a "misleading" argument

Reveal Answer **Answer: (B)** Matching new claims to existing fact-checked claims is well-suited to automation because it is essentially a similarity search problem: given a large index of pre-computed claim embeddings with associated verdicts, a new claim can be matched to similar existing claims in near-real-time using semantic similarity metrics. This scales what human fact-checkers have already done. Options (A) and (D) require genuine knowledge and reasoning that current NLP systems lack. Option (C) requires understanding authorial intent and cultural context — precisely where current models struggle most. Satire has the same surface-level linguistic features as misinformation (sensational claims, emotional language) but different intent, which is not recoverable from text alone.

Question 2 (T/F): A misinformation detection model that achieves 92% accuracy on a benchmark dataset is safe to deploy for automated content removal at scale.

Reveal Answer **Answer: False** High accuracy on a benchmark dataset does not establish safety for large-scale deployment for several reasons: 1. **Scale**: At 92% accuracy with 10 million posts/day, approximately 800,000 posts per day would be incorrectly classified — including many false positives (legitimate speech removed). 2. **Benchmark vs. deployment distribution**: Models trained on benchmark datasets frequently underperform dramatically on deployment data from different domains, time periods, and demographics. 3. **Adversarial robustness**: Benchmark accuracy does not measure robustness to adversarial attacks. 4. **Disaggregated performance**: Overall 92% accuracy may conceal much lower accuracy on specific topics, languages, or communities. 5. **Label quality**: 92% accuracy relative to noisy benchmark labels does not guarantee 92% accuracy relative to ground truth. Safe deployment requires disaggregated evaluation, adversarial testing, human oversight mechanisms, and ongoing monitoring — not just a single accuracy figure.

Question 3 (M): The European Union's Digital Services Act (2022) requirements for automated content moderation reflect which principle?

(A) Automated systems should replace human moderation for efficiency (B) Content moderation decisions should be invisible to users to prevent gaming (C) Meaningful human oversight and contestability of algorithmic decisions is required (D) All misinformation must be removed within 24 hours of detection

Reveal Answer **Answer: (C)** The Digital Services Act requires that platforms: (1) provide transparency about how recommender systems and content moderation work, (2) offer users meaningful options to contest algorithmic decisions, (3) provide access to human review for challenged decisions, and (4) conduct algorithmic impact assessments for very large platforms. This reflects the principle that automated systems should augment and be accountable to human judgment, not replace it. The DSA represents the most comprehensive regulatory framework for platform accountability in this space and has significant implications for how automated misinformation detection is deployed.

Section 2: Text Preprocessing

Question 4 (M): Which preprocessing technique maps "running", "runs", "ran" to "run" using part-of-speech information and a linguistic lexicon?

(A) Stemming (B) Lemmatization (C) Tokenization (D) Case normalization

Reveal Answer **Answer: (B)** Lemmatization uses vocabulary knowledge and morphological analysis — including part-of-speech tagging — to return the canonical dictionary form (lemma) of a word. The words "running," "runs," and "ran" are all forms of the lemma "run." Lemmatization correctly handles irregular forms that rule-based stemming cannot: "ran" → "run" (not "ran"), "better" → "good" (not "better"). Stemming (A) uses rule-based suffix stripping without linguistic knowledge, producing approximate roots that may not be real words ("studies" → "studi"). Tokenization (C) splits text into tokens. Case normalization (D) converts to lowercase.

Question 5 (T/F): For misinformation detection tasks, all words that appear in standard English stopword lists should always be removed from text before feature extraction.

Reveal Answer **Answer: False** Standard stopword lists include function words that carry meaningful signal for misinformation detection and should NOT be removed. Critical words often included in stopword lists include: - **Negation**: "not," "never," "no" — flipping the meaning of adjacent content - **Quantifiers**: "all," "every," "always" — indicating absolutist claims - **Hedging**: "might," "could," "allegedly," "reportedly" — indicating epistemic uncertainty - **Comparison**: "more," "less," "better," "worse" — required for comparative claims Removing "not" causes "vaccines do not cause autism" to have the same features as "vaccines cause autism." Stopword removal must be applied selectively and with awareness of the task's linguistic requirements. Task-adaptive stopword selection using mutual information between words and class labels is preferable to blanket removal.

Question 6 (SA): Explain the difference between word tokenization and subword tokenization. Why do transformer models like BERT use subword tokenization?

Reveal Answer **Sample Answer:** **Word tokenization** splits text into whole-word tokens, typically at whitespace and punctuation boundaries. The vocabulary is fixed: any word not in the pre-specified vocabulary is out-of-vocabulary (OOV) and cannot be represented. **Subword tokenization** (e.g., BERT's WordPiece, GPT's BPE) splits words into frequent subword units. "misinformation" might become ["mis", "##information"]; "vaccinate" might become ["vaccin", "##ate"]. The ## marker indicates continuation from a previous token. **Why transformers use subword tokenization:** 1. **OOV handling**: Subword tokenization can represent any word by decomposing it into known subword units, eliminating the OOV problem. This is essential for social media text with creative spelling, neologisms, and portmanteaus. 2. **Vocabulary efficiency**: Sharing subword units across morphologically related words (e.g., "vaccinated," "vaccinating," "vaccination") reduces vocabulary size while preserving morphological information. 3. **Cross-lingual representation**: Subwords shared across languages can enable multilingual models. 4. **Handling of rare words**: A rare word that would be OOV in word tokenization can be represented through its common subword components. The trade-off is that subword tokenization increases sequence length (more tokens per sentence), which has computational implications for transformer models whose complexity scales quadratically with sequence length.

Section 3: Feature Engineering

Question 7 (M): TF-IDF gives HIGH weight to a term when:

(A) The term appears frequently in many documents across the collection (B) The term appears frequently in a specific document but rarely across the collection (C) The term appears rarely in a specific document but frequently across the collection (D) The term has high document frequency and low term frequency

Reveal Answer **Answer: (B)** TF-IDF = TF(t,d) × IDF(t,D): - **TF(t,d)** is high when the term appears frequently in document d - **IDF(t,D)** = log(N/df_t) is high when df_t (number of documents containing t) is LOW — i.e., when the term is rare across the collection Therefore TF-IDF is highest for terms that are frequent within a specific document AND rare across the overall collection. Such terms are highly distinctive for that document. Common words like "the", "is", "and" have high TF but very low IDF (they appear in almost every document), so their TF-IDF is near zero despite high frequency. Distinctive content words specific to a document's topic get the highest TF-IDF scores.

Question 8 (T/F): Stylometric features (ALL CAPS rate, exclamation mark frequency, readability score) are theoretically grounded predictors of misinformation because deceptive content must by definition use sensationalized language.

Reveal Answer **Answer: False** Stylometric features are empirically predictive — they correlate with low credibility in many datasets — but they are not theoretically guaranteed predictors of falseness. Several important counter-examples: 1. **Sophisticated misinformation** deliberately mimics the calm, professional style of credible journalism to appear authoritative. State-sponsored disinformation is often stylistically indistinguishable from legitimate news. 2. **Credible sensational news**: Genuine breaking news about disasters, crimes, or shocking political events may use emotional language and exclamation marks appropriately. 3. **Satire**: Satirical content uses the same stylistic features as misinformation — hyperbole, absurdist claims, emotional intensity — but is not deceptive. 4. **Technical misinformation**: Health or financial misinformation targeting educated audiences may use complex, technical language with low emotional content. Stylometric features are useful signals that improve classifier performance on many datasets, but they are not definitionally related to falseness and will fail systematically on misinformation that deliberately avoids these patterns.

Question 9 (M): The LIAR dataset's six-class truth labeling scheme (pants-fire, false, barely-true, half-true, mostly-true, true) creates which primary challenge for machine learning models?

(A) Insufficient training examples for any class to learn from (B) The need to distinguish categories with subtle, graduated semantic differences (C) The presence of unlabeled examples that confuse training (D) Class labels that are mutually exclusive, preventing multi-label learning

Reveal Answer **Answer: (B)** The six-way labeling creates the challenge of distinguishing claims that differ in subtle degrees of truthfulness — "barely-true," "half-true," and "mostly-true" require nuanced judgment that is difficult to operationalize from text features alone. These boundary cases involve the distinction between a claim that is technically accurate but missing important context (half-true) versus one that accurately reflects the main point despite minor inaccuracies (mostly-true). The consequence is that state-of-the-art models achieve only ~23–28% six-class accuracy on LIAR — not because they fail to learn anything, but because the task requires fine-grained reasoning that extends well beyond stylistic pattern matching. This illustrates the gap between what NLP models can currently do and what genuine fact evaluation requires.

Section 4: Classical ML Approaches

Question 10 (M): Naive Bayes is called "naive" because:

(A) It was developed before sophisticated statistical theory was available (B) It assumes all features (words) are conditionally independent given the class (C) It uses a naive random initialization rather than principled weight setting (D) It naively rounds probabilities to 0 or 1 rather than maintaining distributions

Reveal Answer **Answer: (B)** Naive Bayes assumes that the probability of observing each feature (word) given the class is independent of all other features. For text classification, this means assuming the probability of the word "vaccine" given a document is "true news" is independent of whether the word "safety" appears in the same document — a mathematically unrealistic assumption (words strongly co-occur based on topic and style). Despite this "naive" assumption, Naive Bayes performs surprisingly well for text classification because even when the independence assumption is violated, the relative ranking of class probabilities is often preserved, leading to correct classifications. The assumption's violation does affect calibration (the probability estimates themselves are poorly calibrated) but less often the most-probable class assignment.

Question 11 (T/F): A Support Vector Machine with a linear kernel is generally more appropriate than a polynomial or RBF kernel for TF-IDF text classification because text features are already high-dimensional.

Reveal Answer **Answer: True** TF-IDF feature vectors are high-dimensional (vocabulary size × 1, often 10,000–100,000 dimensions) and sparse. In high-dimensional spaces, text classification problems are often linearly separable — the linear kernel SVM can find a good separating hyperplane directly in the original feature space without the computational cost of kernel transformations. Non-linear kernels (RBF, polynomial) implicitly map features to even higher-dimensional spaces, which is unnecessary when the linear kernel already works well and adds computational cost. Empirically, LinearSVC (linear kernel SVM optimized for large sparse feature matrices) consistently outperforms non-linear kernels on standard text classification benchmarks. For low-dimensional features derived from text (e.g., stylometric features, embeddings), non-linear kernels may become competitive.

Question 12 (SA): Explain why Random Forest typically performs worse than linear SVM on TF-IDF features but may perform comparably or better when mixed features (text + metadata) are used.

Reveal Answer **Sample Answer:** **Why RF underperforms SVM on pure TF-IDF:** Random Forests build decision trees, each of which recursively partitions the feature space. Decision trees split one feature at a time, which is inefficient for high-dimensional sparse TF-IDF spaces where most features are zero for most documents. The random feature subsampling at each split (typically √(n_features) features) means that with 50,000 features, each split considers only ~224 features — and for a given document (mostly zeros), even fewer will be informative. SVM with linear kernel, by contrast, finds a global optimal hyperplane using all features simultaneously — highly efficient for high-dimensional, sparse data. **Why RF performs better with mixed features:** When features include stylometric measures, metadata (speaker credibility, publication date, category), and dimensionality-reduced text features (e.g., PCA of TF-IDF), the feature space becomes: 1. Lower dimensional (more manageable for tree splits) 2. Heterogeneous (mixing continuous and sparse features that trees handle naturally) 3. Non-linearly separable (requiring the complex decision boundaries that ensembles provide) In this setting, Random Forest's ability to model non-linear interactions between features (e.g., "low readability AND high emotional language AND low-credibility publisher → fake") becomes an advantage. The ensemble averaging also reduces variance compared to single trees.

Section 5: Word Embeddings

Question 13 (M): Given trained Word2Vec embeddings, the operation vector("Paris") - vector("France") + vector("Germany") is expected to produce a vector close to:

(A) vector("Berlin") (B) vector("Europe") (C) vector("city") (D) vector("French")

Reveal Answer **Answer: (A)** This is the word analogy task: "Paris is to France as _______ is to Germany." The answer is "Berlin" — Germany's capital. The vector arithmetic captures the "capital city of country" relationship: subtracting France's vector removes the "France" concept, adding Germany's vector adds the "Germany" concept, and the result is close to the vector representing Germany's capital. This analogy arithmetic works because Word2Vec encodes relational structure in the geometry of the embedding space. The direction from country vectors to their capital city vectors is relatively consistent across the embedding space, so arithmetic operations can navigate this relational structure.

Question 14 (T/F): FastText can generate embeddings for words that were not present in the training vocabulary, which Word2Vec cannot.

Reveal Answer **Answer: True** Word2Vec (and GloVe) represent each word as a single learned vector. Words not in the training vocabulary have no representation — they are out-of-vocabulary (OOV). FastText represents each word as the sum of its character n-gram embeddings (e.g., subwords of length 3–6 characters). Because character n-grams are much more numerous and shared across words, FastText can generate an embedding for any novel word by summing the embeddings of its subword components, even if the whole word was never seen during training. This is particularly valuable for: - Misspellings and typos ("vaccin8" → subwords overlapping with "vaccine") - Social media neologisms and hashtags ("#FakeNews" has subwords in common with "Fake" and "News") - Morphological variations and compound words - Foreign words that share character patterns with known vocabulary

Question 15 (M): Which of the following BEST describes cosine similarity between two document embeddings?

(A) The Euclidean distance between their embedding vectors (B) The dot product of their embedding vectors normalized by their magnitudes (C) The fraction of vocabulary terms they share (D) The correlation between their TF-IDF weight distributions

Reveal Answer **Answer: (B)** Cosine similarity = (A · B) / (||A|| × ||B||) It measures the cosine of the angle between two vectors: 1.0 means identical direction (maximum similarity), 0 means orthogonal (no similarity), -1 means opposite directions (maximum dissimilarity). The normalization by magnitude makes cosine similarity insensitive to document length — a short and long document discussing the same topic will have similar cosine similarity if their content is similar, even though their raw embedding magnitudes differ. This makes cosine similarity more appropriate for document comparison than raw dot products (option B without normalization) or Euclidean distance (option A), which are affected by magnitude/length.

Section 6: Transformer Models

Question 16 (M): BERT's Masked Language Modeling (MLM) pre-training task is designed to:

(A) Train the model to predict the next word in a sequence from left to right (B) Train the model to predict randomly masked tokens using bidirectional context (C) Train the model to classify whether two sentences are consecutive (D) Train the model to translate between languages

Reveal Answer **Answer: (B)** In Masked Language Modeling, approximately 15% of input tokens are randomly replaced with a [MASK] token, and the model is trained to predict the original tokens from the surrounding bidirectional context — both the tokens to the left AND to the right of the masked position. This contrasts with GPT's causal language modeling (option A), which predicts each token from only the left context (prior tokens). By training with bidirectional context, BERT learns rich contextual representations that incorporate information from both directions — enabling it to build representations that reflect a word's full sentential context, not just what precedes it. Option (C) describes BERT's second pre-training task, Next Sentence Prediction (NSP), which was later found to be less important than MLM by RoBERTa's ablation studies.

Question 17 (T/F): Fine-tuning BERT on the LIAR dataset typically produces substantially better performance than TF-IDF + linear SVM on the six-class classification task.

Reveal Answer **Answer: False** On the LIAR six-class classification task, BERT fine-tuning typically achieves 23–28% accuracy — only marginally better than linear SVM with TF-IDF features (24–28%) and roughly in the same range as Naive Bayes + metadata features. This is a well-documented finding in the literature: on LIAR specifically, the gains from transformers over classical methods are modest because: 1. LIAR has only ~10,000 training examples — too few to exploit BERT's full potential 2. The task requires reasoning beyond surface text patterns that BERT alone cannot solve 3. Speaker metadata and credibility features, which classical methods can incorporate easily, are particularly informative for LIAR 4. The six-class granularity is genuinely difficult even for humans BERT shows larger gains over classical methods on FEVER and other tasks where the evidence retrieval component can leverage BERT's stronger semantic understanding.

Question 18 (SA): Explain what "self-attention" does in a transformer model. Why is it more powerful than the recurrence mechanism used in earlier RNN-based sequence models?

Reveal Answer **Sample Answer:** **Self-attention** allows each position in a sequence to directly attend to all other positions simultaneously, computing a weighted combination of all positions' representations based on their relevance to the current position. For each position, the attention mechanism computes: 1. A **query** vector representing "what information am I looking for?" 2. **Key** vectors for all positions representing "what information do I contain?" 3. **Value** vectors for all positions representing the actual information content 4. Attention weights via softmax(QKᵀ/√d_k), indicating which positions are most relevant 5. The output as a weighted sum of value vectors **Advantages over RNNs:** 1. **Parallelization**: RNNs process sequences sequentially — each step depends on the previous. Self-attention computes all positions in parallel, enabling massive speedup with GPU acceleration. 2. **Long-range dependencies**: RNNs must propagate information through many sequential steps to connect distant tokens, leading to vanishing gradient problems over long sequences. Self-attention connects any two positions in O(1) steps, directly modeling arbitrarily long-range dependencies. 3. **No fixed-size bottleneck**: RNN encoders compress the entire sequence into a fixed-size hidden state. Self-attention maintains full sequence information throughout, with each position attending directly to any other. 4. **Interpretability**: Attention weights provide a (limited) window into which tokens are being related to which others, which is not easily available from RNN hidden states.

Section 7: Claim Verification Pipeline

Question 19 (M): In the FEVER benchmark, a claim labeled "NOT ENOUGH INFORMATION" means:

(A) The claim was too vague for annotators to understand (B) The claim is false but the evidence is insufficient to prove it (C) Sufficient evidence to confirm or refute the claim is not available in Wikipedia (D) The claim was generated incorrectly during dataset construction

Reveal Answer **Answer: (C)** "NOT ENOUGH INFORMATION" (NEI) in FEVER indicates that the claim cannot be verified using Wikipedia as the evidence source — not because the claim is necessarily false, but because Wikipedia does not contain sufficient relevant information. This captures the realistic scenario where a fact-checker encounters a claim that they cannot confirm or refute with available evidence. NEI claims in FEVER make up approximately 20% of the dataset. They are the hardest class to classify correctly because the model must determine that no evidence in Wikipedia is sufficient, rather than simply matching claim components to Wikipedia sentences. Models tend to overpredict SUPPORTED and REFUTED (where explicit textual matches exist) and underpredict NEI (where absence of evidence must be recognized).

Question 20 (T/F): The stance of a document toward a claim (supports, refutes, discusses) is equivalent to the truth label of the document (true, false).

Reveal Answer **Answer: False** Stance and truth are orthogonal dimensions: - A factually accurate document can **support** a false claim (e.g., a news article accurately reporting a false rumor in order to debunk it will "discuss" or even quote the false claim) - A factually accurate document can **refute** a false claim (this is the standard fact-check format) - A document can **support** a true claim — the most typical case in normal reporting - A document discussing a false claim factually (without endorsing it) takes a "discuss" or "neutral" stance toward the claim Stance detection is a component of fact verification: identifying which documents support vs. refute a claim helps aggregate evidence for verdict prediction. But a document's stance (how it relates to a specific claim) is not the same as the document's overall truthfulness.

Section 8: Limitations

Question 21 (M): Which of the following is the BEST description of "label leakage" in fake news datasets?

(A) When model performance leaks between training and test sets due to insufficient data splitting (B) When models exploit spurious correlations in dataset construction artifacts rather than learning to detect genuine misinformation (C) When fact-checker labels are biased toward particular political viewpoints (D) When model confidence scores are miscalibrated relative to true accuracy

Reveal Answer **Answer: (B)** Label leakage in fake news datasets refers to cases where models achieve high accuracy by exploiting features that correlate with labels due to dataset construction choices rather than because those features genuinely indicate misinformation. Examples: - If all "fake" articles come from a specific set of publisher domains and the model learns domain name features, it is leaking from domain rather than learning content-based misinformation signals - If crowdworkers generating false claims for FEVER use certain sentence patterns that real Wikipedia sentences don't have, a model can learn those annotation artifacts rather than semantic falseness - If "fake" articles in a dataset are systematically shorter, older, or from a particular time period, length/date features become spurious predictors Label leakage causes models to achieve impressive benchmark performance while failing badly on real-world deployment data where the leaking features don't have the same statistical relationships to labels.

Question 22 (T/F): Adversarial attacks on text classifiers require highly specialized technical knowledge to execute, making them an unlikely threat in real-world misinformation deployment scenarios.

Reveal Answer **Answer: False** Simple adversarial attacks on text classifiers require minimal technical knowledge and have been observed in real-world misinformation campaigns: 1. **Character substitution**: Replacing letters with look-alike characters (using "0" for "o", using homoglyphs, adding zero-width Unicode characters) defeats tokenizers and word-matching classifiers. This is trivial to execute. 2. **Synonym substitution**: Replacing flagged keywords with synonyms or paraphrases — "the election was stolen" → "the election was rigged" → "the vote was fraudulent." No technical knowledge required. 3. **Platform-specific evasions**: On image-based platforms, embedding text in images to evade text classifiers — a technique widely documented in hate speech evasion. 4. **Strategic spreading**: Splitting a false claim across multiple posts, each of which appears benign independently. More sophisticated attacks (gradient-based adversarial examples) do require technical knowledge, but effective evasion of real-world misinformation classifiers has been documented using simple manual techniques by ordinary users, journalists testing platform moderation, and documented disinformation campaigns.

Section 9: Ethics

Question 23 (M): A content moderation system has 95% precision and 80% recall at detecting misinformation. This means:

(A) 95% of flagged posts are actually misinformation; 80% of all misinformation is flagged (B) 80% of flagged posts are actually misinformation; 95% of all misinformation is flagged (C) 95% of legitimate content is correctly passed through; 80% of misinformation is correctly removed (D) The system is correct 95% of the time on average

Reveal Answer **Answer: (A)** - **Precision** = True Positives / (True Positives + False Positives) = fraction of flagged items that are actually misinformation. At 95% precision: 5% of flagged items are false positives (legitimate speech incorrectly flagged). - **Recall** = True Positives / (True Positives + False Negatives) = fraction of all true misinformation that is flagged. At 80% recall: 20% of actual misinformation escapes detection (false negatives). At scale: if a platform processes 1 million misinformation posts and 10 million legitimate posts, the system would flag approximately 800,000 true misinformation posts and ~40,000 false positives (legitimate posts incorrectly flagged), while missing 200,000 actual misinformation posts. Depending on the harm of each type of error and the scale of deployment, different precision-recall trade-offs may be appropriate.

Question 24 (SA): What is the "power concentration problem" in automated misinformation detection, and why does it represent a structural risk that technical solutions alone cannot address?

Reveal Answer **Sample Answer:** The **power concentration problem** refers to the fact that deploying automated misinformation detection at scale concentrates the power to shape public discourse in the hands of a small number of actors: platform operators who deploy the systems, governments that regulate them, researchers whose datasets define "false," and AI companies whose models implement the classification. **Why it is structural:** Every misinformation detection system embeds a definition of what counts as false, misleading, or harmful. That definition reflects the values, knowledge, cultural assumptions, and political perspectives of the people who created the datasets, wrote the annotation guidelines, and built the models. When those systems operate at global scale — affecting billions of posts across hundreds of countries — the embedded assumptions are applied universally, regardless of whether they are appropriate in specific cultural, political, or epistemic contexts. **Why technical solutions are insufficient:** No amount of model improvement eliminates this structural problem because the issue is not technical accuracy but epistemic and political authority: - A 99% accurate model is still making the embedded definitional choices with 99% efficiency - Improving detection of coordinated inauthentic behavior still requires someone to define what "authentic" behavior means - Making models fairer within a given definition of misinformation does not address whether that definition itself is appropriate Addressing the power concentration problem requires institutional and legal mechanisms: independent oversight bodies, regulatory requirements for transparency and accountability, democratic deliberation about the norms governing public speech, and structural separation between platform operators' financial interests and content moderation decisions.

Question 25 (M): Which of the following would MOST effectively address the concern that automated content moderation systems disproportionately affect certain linguistic or demographic communities?

(A) Increasing the overall accuracy of the model from 90% to 95% (B) Disaggregating evaluation metrics by community, language variety, and topic before deployment (C) Increasing the size of the training dataset (D) Using a more complex model architecture

Reveal Answer **Answer: (B)** Disaggregated evaluation — measuring performance separately for different demographic groups, languages, dialects, and topics — is the most direct way to identify differential error rates before deployment. Overall accuracy can be high while a specific community experiences disproportionately high false positive rates (a documented phenomenon in both facial recognition and content moderation systems). Options (A), (C), and (D) might improve overall performance metrics but do not specifically address disparate impact. A model that is 5% more accurate overall but still systematically misclassifies African American Vernacular English is more powerful but not more equitable. Disaggregated evaluation is a core recommendation of algorithmic fairness research (the "Model Cards" framework, NIST AI RMF, and the EU AI Act all include provisions for this). It requires collecting demographic data and designing evaluation sets that represent affected communities — a significant but necessary investment before deployment.