Case Study 2: AI in the Headlines — Separating Signal from Noise

The Challenge of AI Journalism

You are scrolling through your news feed on a Tuesday morning. In the span of five minutes, you encounter the following headlines:

Headline A: "AI System Achieves 'Superhuman' Performance in Medical Diagnosis"

Headline B: "Experts Warn: AI Could Eliminate 40% of Jobs Within 15 Years"

Headline C: "AI Passes the Bar Exam — Is the Legal Profession Doomed?"

Headline D: "Company Claims Its AI Can Predict Criminal Behavior Before It Happens"

Each of these headlines is designed to provoke a reaction — excitement, fear, awe, alarm. Each one contains a grain of truth embedded in layers of framing, omission, and implication. Your job, as a developing AI-literate reader, is to separate the signal from the noise.

In this case study, we will walk through each headline using the FACTS Framework and the concepts from Chapter 1. By the end, you will have practiced a skill you will use for the rest of your life: reading about AI without being manipulated by the writing.

Headline A: "AI System Achieves 'Superhuman' Performance in Medical Diagnosis"

What is probably true: Research teams have developed AI systems that match or exceed human expert performance on specific diagnostic tasks. For example, a 2020 study published in Nature (McKinney et al.) found that an AI system outperformed radiologists in breast cancer screening using mammograms in a controlled study setting.

What the headline obscures:

  • Specificity. The AI did not achieve superhuman performance in "medical diagnosis" broadly. It achieved it on one specific task (e.g., detecting a particular type of cancer in a particular type of image). The headline collapses a narrow achievement into a sweeping claim.

  • Context of the test. Performance in a controlled research study often does not translate directly to real-world clinical settings, where images are messier, patient populations are more diverse, and the consequences of errors are immediate. This is sometimes called the "lab-to-clinic gap."

  • The word "superhuman." This word implies the AI is categorically better than humans. In practice, these studies typically show that AI performs better than humans on average — meaning it outperforms the average radiologist but may not outperform the best ones. And the AI makes different kinds of errors than humans, which matters for patient safety.

  • Who was in the study population? If the system was trained and tested primarily on images from patients of European descent, its "superhuman" performance may not generalize to other populations.

FACTS check (abbreviated): - F: What specific diagnostic task? (Not "all of medicine.") - A: Accuracy compared to whom — average or expert clinicians? In what conditions? - C: If deployed prematurely, who is helped and who is put at risk? - T: What patient populations were represented in the training data? - S: If a "superhuman" AI misses a diagnosis, does the hospital or the AI vendor bear responsibility?

Headline B: "Experts Warn: AI Could Eliminate 40% of Jobs Within 15 Years"

What is probably behind this: Several widely cited studies have estimated the percentage of jobs "exposed to" AI automation. A frequently referenced 2023 report by Goldman Sachs estimated that generative AI could affect roughly 300 million full-time jobs globally — a figure that was widely reported as "AI will eliminate" those jobs, which is not what the report said.

What the headline obscures:

  • "Exposed to" vs. "eliminated." Most serious research distinguishes between jobs that are exposed to AI automation (meaning some tasks within the job could be automated) and jobs that will be entirely eliminated. These are vastly different claims. A job where 30% of tasks can be automated is a changed job, not a lost one.

  • "Could" is doing heavy lifting. The word "could" makes the headline technically defensible while implying certainty. "AI could eliminate 40% of jobs" and "AI could eliminate 0% of jobs" are both true statements — "could" covers any possibility.

  • "Experts" is vague. Which experts? From which fields? With what assumptions? Labor economists, AI researchers, and technology executives often make very different predictions based on different models and different assumptions about adoption speed, regulation, and human adaptability.

  • Historical context. Previous waves of automation (the steam engine, electricity, the personal computer, the internet) also generated predictions of mass unemployment. In each case, technology eliminated some jobs, created others, and transformed many. This does not mean AI will follow exactly the same pattern, but it does mean that simple extrapolation from current trends is unreliable.

Key lesson: When a headline uses the word "could" paired with a dramatic number, it is a signal to slow down and ask what the underlying research actually claims.

What is probably true: In early 2023, OpenAI reported that GPT-4 scored in approximately the 90th percentile on the Uniform Bar Examination. This was a genuine technical achievement, demonstrating remarkable language processing capabilities.

What the headline obscures:

  • Passing a test is not the same as doing a job. The bar exam tests legal knowledge and analytical writing in a controlled format. Practicing law involves client relationships, courtroom judgment, ethical reasoning under ambiguity, negotiation, understanding unspoken social dynamics, and much more. An AI that passes the bar exam is not an AI that can practice law — just as a student who aces a written driving test is not necessarily a good driver.

  • The question assumes a binary. "Is the legal profession doomed?" implies that AI will either destroy the profession or leave it untouched. The actual outcome is almost certainly in between: AI tools will change how legal work is done (automating document review, accelerating research, drafting routine filings) while leaving many aspects of legal practice — judgment, advocacy, ethical reasoning — firmly in human hands.

  • Narrow vs. general AI, again. The AI that passed the bar exam is narrow AI — it is very good at processing and generating text. It does not "understand" law. It cannot assess whether a client is telling the truth, navigate a tense plea negotiation, or decide whether to take a case on moral grounds.

FACTS check (abbreviated): - F: It passed a standardized test, not practiced law. - A: 90th percentile on the bar exam — but what about tasks that are not on the exam? - C: Legal assistants and paralegals may be more affected than senior attorneys. Who in the profession is most vulnerable? - T: Trained on vast text data — but legal practice involves privileged and confidential information. What are the data governance implications?

Headline D: "Company Claims Its AI Can Predict Criminal Behavior Before It Happens"

What is probably behind this: Several companies have developed or marketed "predictive" tools for law enforcement. Some predict where crimes are likely to occur (place-based prediction); others claim to predict which individuals are likely to commit crimes (person-based prediction). The distinction between these two approaches is enormous, but headlines often collapse them.

What the headline obscures:

  • "Predict criminal behavior" is deeply misleading. What these systems actually do is identify statistical correlations in historical data. They do not predict the future; they extrapolate from the past. If the past data is biased (and crime data almost always is), the predictions will replicate those biases.

  • The feedback loop problem. This connects directly to CityScope Predict from the chapter. If a system predicts more crime in a neighborhood, more police are sent there, more arrests are made, the data shows more crime, and the system's prediction is "confirmed" — even if the underlying crime rate is no different from other neighborhoods.

  • The presumption of innocence. Person-based prediction raises profound civil liberties concerns. Flagging individuals as likely future offenders — before they have done anything — conflicts with foundational principles of justice. People who are flagged may face increased surveillance, more frequent stops, or harsher treatment, all based on statistical probability rather than evidence of wrongdoing.

  • "Company claims" should be a red flag. The claims come from the company selling the product. They have a financial incentive to overstate the system's capabilities. Independent, peer-reviewed validation is a minimum threshold for credibility, and many predictive policing tools have not met it.

Key lesson: When a company makes claims about its own AI product, apply the same skepticism you would to any advertisement. Ask: Has this been independently validated? By whom? With what methodology?

Patterns Across All Four Headlines

Now that we have examined each headline, step back and notice what they have in common:

  1. Narrow achievements are framed as sweeping breakthroughs. A system that excels at one specific task becomes "AI achieves superhuman performance." The scope of the claim expands far beyond the scope of the evidence.

  2. Nuance is sacrificed for drama. "Some tasks within some jobs may be partially automated over several decades" does not get clicks. "AI will eliminate 40% of jobs" does.

  3. The word "AI" functions as a black box. Each headline uses "AI" as if it refers to a single, unified technology. In reality, the systems described use different techniques, are designed for different purposes, and have different limitations.

  4. Who benefits from the framing is worth asking. Dramatic headlines benefit media outlets (engagement), tech companies (investment and hype), and sometimes researchers (funding and attention). The public may not benefit from distorted information.

  5. The FACTS Framework catches what casual reading misses. In every case, slowing down and asking the five FACTS questions reveals gaps, assumptions, and unstated implications that the headline conceals.

Discussion Questions

  1. Media literacy meets AI literacy. How is evaluating AI claims similar to evaluating political claims or health claims in the media? What skills transfer, and what is unique about AI?

  2. The incentive problem. Journalists need clicks; companies need investment; researchers need funding. How do these incentives shape the way AI is presented to the public? Can you think of structural changes (not just individual behavior changes) that could improve the quality of AI reporting?

  3. Your own experience. Find a recent AI-related headline from a source you normally read. Apply the FACTS Framework and the patterns identified in this case study. What does the headline obscure? What follow-up questions would you want answered?

  4. The responsibility of language. The headline "AI Passes the Bar Exam" uses the word "passes" — a word we normally apply to humans. How does the choice of language shape public understanding of AI capabilities? Can you rewrite any of the four headlines to be more accurate without being boring?

  5. Philosophical dimension. Headline D (predicting criminal behavior) raises the question: Should we use statistical tools to predict individual behavior even if the predictions are accurate? Is there a moral difference between predicting where crime might occur and predicting who might commit it?

Reflection Exercise

Choose one of the four headlines and write two versions of it:

  • Version 1: A headline that is equally attention-grabbing but more accurate.
  • Version 2: A headline that is fully accurate, even if it is less exciting.

Then, in 100–150 words, reflect on the tension between accuracy and engagement in science and technology reporting. Is it possible to be both honest and interesting? What would it take?