Chapter 37 Quiz

Multiple Choice

1. In the ODA speeches dataset, the text_excerpt column is used for feature computation rather than full_text because: - A) text_excerpt is more accurate than full_text - B) full_text is null for approximately 23% of records, while text_excerpt is available for all records - C) NLP models perform better on shorter texts - D) The full_text column contains personal information that should not be analyzed

Answer: B. Using text_excerpt ensures consistent feature computation across all 14,782 speeches. Using full_text would require either excluding 23% of the dataset or imputing for null records, both of which introduce additional methodological challenges.

2. The term_density function in the chapter uses sentence-level presence (whether any sentence contains the term) rather than raw word frequency because: - A) Sentence-level presence is computationally faster than word frequency - B) This follows the Rooduijn-Pauwels approach, which counts whether a sentence contains populist language rather than how many times terms appear - C) Word frequency counting is not possible in Python - D) Terms that appear multiple times in one sentence are less important than terms spread across multiple sentences

Answer: B. The Rooduijn-Pauwels PEA-adapted text analysis approach measures sentence-level density to avoid one sentence with repeated terms dominating the score. The question is: in how many sentences does populist language appear?

3. The chapter's classifier achieves an AUC-ROC of approximately 0.79–0.82. What does this mean in practice? - A) The classifier correctly classifies 79–82% of all speeches - B) The classifier can distinguish populist from non-populist speeches with 79–82% probability when presented with a randomly selected pair (one from each class) - C) 79–82% of populist speeches are correctly identified as populist - D) The classifier makes errors on 18–21% of all speeches

Answer: B. AUC-ROC measures discrimination ability: the probability that a randomly selected positive (populist) speech will receive a higher predicted probability than a randomly selected negative (non-populist) speech. It is not the same as accuracy.

4. The chapter argues that the classifier's false negatives are concentrated among "sophisticated populist communicators." What does this mean technically? - A) The classifier assigns very low probability to speeches that expert human coders would classify as highly populist - B) The classifier assigns very low probability to speeches that score high on the existing populism_score - C) The classifier performs poorly on speeches from politicians who avoid explicit anti-elite vocabulary while still using narrative and indirect framing to achieve populist effects - D) The classifier cannot process speeches longer than a certain word count

Answer: C. False negatives in the context of the populist classifier are speeches that are genuinely populist but classified as non-populist by the model. This happens most with sophisticated communicators who achieve populist effects through story, implicit framing, and indirect language rather than the explicit vocabulary terms in the feature dictionaries.

5. When the chapter computes plural_pronoun_ratio — the ratio of first-person-plural to total first-person pronouns — what aspect of populist rhetoric is it designed to capture? - A) The proportion of the speech focused on group activities rather than individual activities - B) The rhetorical subordination of the leader to "the people" by emphasizing collective "we" over individual "I" - C) The reading level and accessibility of the speech - D) Whether the speech is addressing multiple groups simultaneously

Answer: B. High first-person-plural relative to first-person-singular reflects the populist rhetorical move of foregrounding collective identity ("we the people") over individual leadership ("I will do X"). This creates the impression that the leader speaks as part of the people, not above them.

6. The chapter recommends using logistic regression as the final classifier rather than gradient boosting despite similar AUC scores. The primary reason is: - A) Logistic regression trains faster - B) Logistic regression requires less data - C) Logistic regression coefficients are directly interpretable as feature importance, which is essential for understanding what linguistic patterns matter most - D) Logistic regression has higher accuracy on text classification tasks generally

Answer: C. For a research application where the goal is not just to classify speeches but to understand what makes them populist, model interpretability is as important as predictive accuracy. Logistic regression coefficients directly answer "which features predict populism most strongly?" in a way that tree ensemble importances only partially capture.

7. Sam Harding's methodology statement includes the line: "What this classifier does NOT measure: whether the populism is 'left' or 'right' variant." This is because: - A) The features are insufficient to distinguish left from right populism - B) The classifier was not designed to make this distinction; it measures populist structure (people vs. elite) which is shared across variants - C) Left and right populism use identical vocabulary, making distinction impossible - D) The ODA dataset lacks sufficient left-populist speeches for classifier training

Answer: B. This is the thin-ideology framework applied to measurement design. The classifier deliberately measures the structural dimension (people vs. elite, Manichean framing) that both left and right populism share. Left-right distinction requires measuring content (which groups fill the "elite" slot), which requires different features. Design intention, not just capability, shapes what the classifier does and doesn't measure.

8. The chapter finds that rally speeches show higher populism density than floor speeches by the same politicians. The most cautious methodological interpretation of this finding is: - A) Politicians are dishonest populists who perform populism at rallies without believing it - B) Politicians strategically adapt their rhetorical register to different contexts, which is consistent with multiple underlying theories about populism - C) Rally speeches are sampled from more extreme politicians in the dataset - D) The classifier is biased toward detecting populism in informal speech registers

Answer: B. The finding that rhetoric varies by context is consistent with multiple theories: strategic context-adaptation, social desirability effects in formal settings, audience composition effects, and others. The cautious interpretation acknowledges this range of explanations rather than concluding that politicians are hypocritically "performing" populism.

Short Answer

9. The contrast_ratio feature uses regular expressions to detect binary framing patterns. Describe two patterns it is designed to detect and explain why each captures a populist rhetorical move.

Model Answer: (1) r'\b(vs\.?|versus)\b' — detects explicit "X vs. Y" constructions. This captures the Manichean framing that divides the world into two opposed camps, which is definitionally central to populism's worldview. "The people vs. the establishment" is a canonical populist construction. (2) r'\bnot\b.{1,40}\bbut\b' — detects "not X but Y" contrast patterns. This captures the rhetorical move of defining oneself or one's allies by contrast with the other group: "Not the experts, but the people"; "Not the donors, but the voters." This creates the same people-elite distinction through a grammatical contrast structure rather than explicit populist vocabulary. Both patterns exploit the structural feature that populism is definitionally oppositional — it defines "the people" partly through contrast with "the elite."

10. The chapter uses class_weight='balanced' in the Random Forest classifier. Explain what this parameter does and why it matters for the populism classification task.

Model Answer: class_weight='balanced' instructs the classifier to weight the minority class (populist speeches, approximately 18-22% of the dataset) more heavily during training, so that errors on populist speeches are penalized more than errors on non-populist speeches. Without this adjustment, a classifier can achieve high accuracy simply by predicting "non-populist" for every speech — because the majority class (non-populist) is much larger. Balancing class weights forces the classifier to actually learn to distinguish populist from non-populist speeches rather than defaulting to the majority class. This matters for the populism task because the class imbalance would otherwise produce a classifier that is technically accurate but practically useless for identifying the populist speeches that are the analytical target.

True/False with Explanation

11. True or False: The full_text column should always be preferred over text_excerpt for populism feature computation when available.

Nuanced answer — partially true. Full-text features will generally be more accurate measures of a speech's overall rhetorical character because they reflect the entire speech rather than a 500-word sample. For speeches where the excerpt is a representative sample, the difference may be small. However, using full_text introduces sample selection bias: the 23% of speeches with null full_text would be excluded from any analysis that requires full-text features, unless you impute or handle missing values carefully. For tracking applications where consistent coverage matters more than precision on individual speeches, using text_excerpt for all speeches may be preferable to using full_text for only 77%. The best practice is to compute features from both where available, compare them, and explicitly report which you used and why.

12. True or False: A classifier that achieves AUC=0.80 on a training/test set can be trusted to identify all populist speeches in a new dataset with 80% accuracy.

False. AUC measures discrimination on the dataset the model was evaluated on, under specific conditions (same class balance, same data distribution, same time period). When applied to a new dataset, the classifier may face: different class balance (different proportion of populist speeches); rhetorical evolution (vocabulary changes in the new period); different speaker composition; different speech types. The temporal generalization exercise (Exercise 37.5) directly tests this concern. AUC on a held-out test set is a measure of expected performance on data similar to the training data, not a guaranteed accuracy level on all future data.