Chapter 37 Exercises

Conceptual Exercises

Exercise 37.1: Threshold Sensitivity Analysis The chapter uses 0.40 as the threshold for creating binary populist/non-populist labels from the populism_score column. Implement the threshold sensitivity analysis shown in exercise-solutions.py (Exercise 37.1) and: - Run classifiers at thresholds from 0.25 to 0.60 in 0.05 increments - Report AUC-ROC and positive class percentage at each threshold - Visualize the results as two-panel plot - Write a 150-word interpretation explaining which threshold you would choose and why

Exercise 37.2: Left vs. Right Populism Classifier Using only the speeches classified as populist (populism_score ≥ 0.40), build a classifier that tries to distinguish Democratic from Republican populist speeches. Compare performance using: - Only the eight structural features from Chapter 37 - Adding party-specific elite vocabulary (see Exercise 37.3 in exercise-solutions.py)

What does the AUC comparison tell you about the thin-ideology framework's prediction that left and right populism share structural features?

Exercise 37.3: Adding New Features The chapter's eight features are all derived from single-token and multi-word expression counting. Design and implement at least two additional features that capture populist communication patterns not covered by the existing eight: - Feature design: What aspect of populist communication are you trying to capture? - Theoretical justification: Which element of the ideational definition does it operationalize? - Implementation: Write the Python function - Evaluation: Does adding the feature improve AUC? Does it correlate with populism_score?

Programming Exercises

Exercise 37.4: Garza Speech Profile If you have access to speech data for a real Democratic Senate candidate, apply the full feature extraction and classification pipeline to their corpus. If not, use the exercise-solutions.py code to analyze the closest available proxy (Democratic Senate candidates generally). Report: - Where their mean feature values fall in the overall corpus percentile distribution - Whether their speeches lean toward people-centric or anti-elite populism (or neither) - Whether their rhetoric is changing over time - How this connects to the strategic choices described in Chapter 34

Exercise 37.5: Temporal Generalization Test Implement the temporal generalization check from exercise-solutions.py (Exercise 37.5): - Train the classifier on speeches from before a cutoff year - Test on speeches from after the cutoff year - Compare temporal AUC to standard cross-validated AUC - Interpret what a significant AUC drop would mean for a researcher using this classifier to track "current" populist rhetoric trends

Exercise 37.6: State-Level Rhetoric Variation Using the ODA speeches dataset, create a state-level analysis of populist rhetoric: - Compute mean populism_score and mean anti_elite_density by state - Create a visualization showing geographic variation - Test whether states with competitive Senate races differ significantly from safe states on rhetoric measures - Interpret the finding in light of the Garza-Whitfield Texas race context

Interpretation Exercises

Exercise 37.7: Sam Harding's Methodology Statement The chapter includes a full methodology statement for the ODA populist rhetoric classifier. Using this as a template, write a methodology statement for your own classifier if you have modified the feature set or model architecture. The statement must: - Define what the classifier measures (with theoretical grounding) - List features explicitly - State appropriate and inappropriate uses - Identify at least three specific limitations

Exercise 37.8: The Map and the Territory Write a 400-word critical reflection on one specific finding from your classifier analysis (or from the chapter's example analyses). Address: - What does the finding actually say (the map)? - What political reality might it correspond to (the territory)? - What political reality is the classifier likely missing (where the map is inaccurate)? - What would a researcher need to do to check whether the map accurately represents the territory?

Advanced Extension

Exercise 37.9: Full-Text Feature Engineering For the approximately 77% of ODA speeches with full_text available, recompute all eight features using the full text rather than the 500-word excerpt. Compare: - Mean feature values: do full texts produce systematically different scores than excerpts? - Classifier performance: does full-text feature extraction improve AUC? - Which features change most between excerpt and full text?

Interpret: If features computed from 500-word excerpts are highly correlated with features computed from full texts, then the excerpt-based approach is adequate for tracking purposes. If they diverge substantially, future work should prioritize full-text collection.

Exercise 37.10: Building a Production Tracker Design (in pseudocode or implementation) a production-level populist rhetoric tracker that: 1. Ingests new speech text from an RSS feed or web scraping pipeline 2. Applies the feature extraction and classification pipeline 3. Flags speeches above a high-confidence threshold for human review 4. Updates a time-series dashboard daily

What are the engineering challenges? What are the methodological risks of fully automating this process without human review?