Case Study: Scoring Other Frameworks — Applying Self-Critique to Popular Analytical Tools

The Exercise

If self-critique is the mark of a trustworthy framework, then every analytical framework should be subject to its own analysis. This case study applies the Red Flag Scorecard to three widely used frameworks — not to debunk them, but to demonstrate the self-critique methodology.

Framework 1: Kuhn's Paradigm Shifts

Central claim: Science progresses through paradigm shifts — periods of "normal science" punctuated by revolutionary changes when anomalies accumulate beyond the paradigm's capacity to absorb them.

Selected Red Flag scores: - Q3 (Falsifiability): 🟡 — What would disprove the paradigm shift model? Since it describes a general pattern, any specific counter-example (a field that changed gradually rather than through revolution) can be accommodated as "not a paradigm shift case." - Q5 (Evidence age): 🟡 — Published in 1962. The core evidence is from the history of physics. Has the model been tested against fields Kuhn didn't examine? - Q8 (Independent sources): 🔴 — The paradigm shift framework comes from a single author and has become so influential that it shapes how subsequent historians describe change — potentially creating a self-fulfilling analytical lens.

Assessment: Kuhn's framework is enormously influential and genuinely insightful. But its very influence may have created an authority cascade in the history and philosophy of science — the framework shapes how people see scientific change, making it difficult to evaluate whether the pattern is really there or whether the framework is imposing it.

Framework 2: Kahneman's System 1 / System 2

Central claim: Human cognition operates through two systems: System 1 (fast, automatic, intuitive) and System 2 (slow, deliberate, analytical). Many cognitive biases result from System 1 overriding System 2.

Selected Red Flag scores: - Q2 (Replication): 🟡 — Some System 1/2 demonstrations replicate well (anchoring, loss aversion); others have been challenged (ego depletion, some priming effects) - Q11 (Simpler explanation): 🟡 — Is the dual-system framework a real description of brain architecture, or a useful metaphor? The neuroscience is more complex than the clean two-system model suggests. - Q8 (Independent sources): 🟡 — The specific framing comes primarily from Kahneman and Tversky's research tradition. Other researchers have proposed different models of dual processing.

Assessment: System 1/2 is a powerful heuristic for understanding cognitive biases. The self-critique reveals that it may be more metaphor than mechanism — a useful way to talk about fast vs. slow cognition rather than a literal description of two brain systems.

Framework 3: Porter's Five Forces

Central claim: Industry profitability is determined by five competitive forces: threat of new entrants, bargaining power of suppliers, bargaining power of buyers, threat of substitutes, and industry rivalry.

Selected Red Flag scores: - Q12 (Prediction track record): 🔴 — The framework is used for analysis, not prediction. It is rarely tested against outcomes — has applying Five Forces analysis actually produced better strategic decisions? - Q10 (Works outside the lab): 🟡 — Developed primarily from case studies of U.S. industries. Less tested in different economic contexts. - Q4 (Who benefits): 🟡 — Strategy consulting firms benefit from the framework's adoption — it creates demand for the analysis they sell.

Assessment: Five Forces is widely taught and used. The self-critique reveals that it has rarely been validated predictively — it is a framework for organizing thinking rather than a validated model of industry dynamics.

The Meta-Lesson

Every framework, including this book's, has vulnerabilities when subjected to its own standards (or to general analytical standards). The purpose of self-critique is not to destroy frameworks but to calibrate confidence in them — to understand what they illuminate and what they obscure, what they explain and what they assume, where they are strong evidence and where they are useful heuristics.

Analysis Questions

1. Choose a framework widely used in your field and apply the full 15-question Red Flag Scorecard to it. What score does it receive? What does the score reveal about the framework's strengths and weaknesses?

2. The three frameworks scored above (Kuhn, Kahneman, Porter) are all enormously influential. Does influence correlate with accuracy, or can a framework become dominant through the same authority cascade and plausible story mechanisms that sustain wrong ideas in other fields?

3. Is there a framework that would score perfectly on the Red Flag Scorecard — all green flags? If not, what does that tell you about the nature of analytical frameworks in general?