Case Study 13.1: The EU AI Act in Practice — Classifying Risk

Background

The EU AI Act entered into force in August 2024, with its provisions phasing in over a multi-year timeline. By 2027, the Act will be fully enforceable, and every AI system deployed in the European Union — or affecting EU residents — will need to be classified by risk level and comply with the corresponding requirements.

But what does risk classification look like in practice? The Act's categories seem clear on paper — unacceptable, high, limited, minimal — but real AI systems do not always fit neatly into boxes. This case study examines how several real-world AI systems would be classified under the Act, revealing the ambiguities and judgment calls that regulators, companies, and courts will need to resolve.

Scenario 1: An AI Hiring Tool

System: RecruitSmart is an AI system used by a multinational corporation to screen job applications. It analyzes resumes, cover letters, and online assessment results to rank applicants and recommend which candidates should advance to interviews.

Classification: High Risk. The EU AI Act explicitly lists "AI systems intended to be used for recruitment or selection of natural persons, in particular to place targeted job advertisements, to analyse and filter job applications, and to evaluate candidates" as high-risk AI systems.

What this means in practice: - RecruitSmart's developer must implement a risk management system — a continuous process of identifying, evaluating, and mitigating risks throughout the system's lifecycle. - The system must be trained on high-quality data that is relevant, representative, and as free from errors as possible. If the training data over-represents certain demographic groups or reflects historical hiring biases, this must be addressed. - The developer must provide technical documentation explaining how the system works, what data it uses, and what its known limitations are. - The deploying company must ensure human oversight — a human being must be able to understand the system's outputs, override its decisions, and intervene when necessary. - The system must undergo a conformity assessment before it can be placed on the market, demonstrating that it meets all the Act's requirements. - Users (the hiring company) and affected persons (job applicants) must receive transparent information about the system's use.

The hard question: RecruitSmart also analyzes candidates' LinkedIn profiles and publicly available social media posts to assess "cultural alignment." Does this additional feature change the risk classification? Not the level — it is already high risk — but it raises additional questions about the data quality and fundamental rights requirements. Analyzing social media could indirectly reveal protected characteristics (religion from church posts, disability from support group membership, pregnancy from congratulatory messages), creating discrimination risks that the risk management system must address.

Scenario 2: A Content Moderation AI

System: ContentGuard, our anchor example, automatically reviews user posts on a social media platform, flagging or removing content that violates the platform's community guidelines.

Classification: This is where it gets interesting. Content moderation AI does not appear on the EU AI Act's explicit list of high-risk AI systems. Under a straightforward reading, it might be classified as limited risk — subject to transparency obligations (users should know that AI is making moderation decisions) but not the full compliance requirements of high-risk systems.

However, the classification is more complex than it first appears:

  • The EU's Digital Services Act (DSA), which works alongside the AI Act, imposes significant obligations on large online platforms regarding content moderation, including transparency reports, complaint mechanisms, and independent audits. ContentGuard would be governed by the DSA's requirements as well as the AI Act's.
  • If ContentGuard uses general-purpose AI models (like large language models) as part of its moderation pipeline, the GPAI provisions of the AI Act would apply to those models, requiring training data transparency and copyright compliance.
  • If ContentGuard makes decisions that significantly affect users' fundamental rights — such as freedom of expression — there is an argument that it should be treated as high risk even if not explicitly listed. The Act includes provisions for the Commission to add new high-risk categories as needed.

The hard question: A platform argues that ContentGuard is not "high risk" because its decisions are about content, not about people — it classifies posts, not persons. A civil liberties organization responds that decisions about what speech is allowed are among the most consequential decisions affecting fundamental rights in a democracy. Who is right? The answer may ultimately come from courts interpreting the Act.

Scenario 3: A Predictive Policing System

System: CityScope Predict, our anchor example, analyzes historical crime data to predict where criminal activity is likely to occur and recommend police patrol patterns.

Classification: This depends critically on how the system works.

If CityScope Predict generates predictions about where crime will occur — analyzing geographic patterns, time-of-day trends, and environmental factors — without profiling or assessing individual people, it would likely be classified as high risk. AI used in law enforcement for "evaluation of the reliability of evidence" or as decision support tools falls into the high-risk category.

If CityScope Predict generates predictions about who will commit crimes — assessing the likelihood that specific individuals will offend based on profiling or personal characteristics — it would be classified as unacceptable risk and banned. The Act explicitly prohibits "AI systems for making risk assessments of natural persons in order to assess or predict the risk of a natural person committing a criminal offence, based solely on the profiling of a natural person."

The hard question: The line between "predicting where" and "predicting who" is blurrier than it sounds. If CityScope Predict predicts that crime will occur in a specific block at a specific time, and that block has only a handful of residents, the geographic prediction effectively becomes a prediction about specific people. If the system's training data encodes racial patterns in policing — as we discussed in Chapter 9 — then "predicting where" may function as a proxy for "predicting who" based on race.

This is exactly the kind of ambiguity that regulators and courts will spend years resolving.

Scenario 4: A Medical Diagnostic AI

System: MedAssist AI, our anchor example, analyzes medical images and patient data to assist doctors in diagnosing diseases.

Classification: High Risk. AI systems "intended to be used as a safety component of a product, or the AI system is itself a product" covered by EU product safety legislation — including medical devices — are classified as high risk. Medical AI is also explicitly listed: AI "intended to be used for the purpose of determining access to or influencing decisions on eligibility, granting, reducing, revoking or reclaiming of health-related services" is high risk.

What this means in practice: MedAssist AI must comply with the full high-risk requirements: risk management, data quality, documentation, transparency, human oversight, accuracy, and cybersecurity. Crucially, it must also comply with existing medical device regulations (the EU Medical Device Regulation, or MDR), meaning it faces two overlapping regulatory frameworks.

The hard question: MedAssist AI uses a deep learning model whose internal decision-making process is not fully interpretable — it can identify a tumor in a scan, but it cannot fully explain why it reached that conclusion. The AI Act requires transparency and explainability for high-risk systems. How do you make a system transparent when even its developers do not fully understand how it reaches its conclusions? This tension between AI capability (deep learning excels at pattern recognition) and regulatory requirements (explainability) is one of the most significant unresolved challenges in AI governance.

Scenario 5: A Generative AI Model

System: A large language model (similar to GPT-4 or Claude) offered as a general-purpose service via API, used by downstream developers for everything from customer service chatbots to legal research tools.

Classification: General-purpose AI model (GPAI) with potentially systemic risk if it exceeds certain capability thresholds.

The model provider must: - Publish a sufficiently detailed summary of training data content - Comply with EU copyright law - Provide technical documentation describing capabilities and limitations

If designated as systemic risk: - Conduct and document adversarial testing ("red-teaming") - Track and report serious incidents - Ensure adequate cybersecurity - Report the model's energy consumption and computational resources

The hard question: A GPAI model is general-purpose — it can be used for minimal-risk applications (writing poetry) and high-risk applications (screening job candidates). The risk depends not on the model itself but on how it is deployed. The AI Act addresses this by placing obligations on both the model provider (transparency, safety testing) and the deployer (compliance with high-risk requirements if used in a high-risk context). But in practice, a model provider cannot control every downstream use. If an employer uses a general-purpose chatbot to screen resumes — a use case the model provider never intended — who is responsible for compliance?

Discussion Questions

  1. Classification ambiguity: Several of these scenarios involve genuine ambiguity about risk classification. Do you think this ambiguity is a fatal flaw in the EU AI Act's approach, or is it an inevitable feature of any regulation that tries to be precise about a diverse technology? How should regulators handle edge cases?

  2. The "where vs. who" distinction: The predictive policing scenario highlights the difference between predicting where crime will occur and predicting who will commit it. Do you think this is a meaningful distinction, or does geographic prediction inevitably become a proxy for individual profiling in practice?

  3. Explainability vs. capability: The MedAssist AI scenario raises the tension between AI performance (deep learning models that achieve high accuracy) and regulatory requirements for explainability (understanding how the system reaches its conclusions). How should this tension be resolved? Is it acceptable to deploy a system that saves lives even if we cannot fully explain how it works?

  4. General-purpose model responsibility: If a general-purpose AI model is used for a high-risk application that its developer never intended, who should bear regulatory responsibility — the model provider, the deployer, or both? How should the law allocate responsibility when the same model can be used for harmless and harmful purposes?

  5. Cross-regulation complexity: MedAssist AI must comply with both the AI Act and the Medical Device Regulation. ContentGuard must comply with both the AI Act and the Digital Services Act. Is this layered approach effective governance or bureaucratic complexity? How should overlapping regulations be coordinated?

Connection to Chapter Themes

This case study demonstrates several key themes from Chapter 13:

  • The definition and classification problem: Real AI systems do not always fit neatly into regulatory categories. The same system might be classified differently depending on how it is used, what data it processes, and how broadly "profiling" is interpreted.
  • The pacing problem: The scenarios described here are based on AI systems that exist today. By the time the AI Act is fully enforceable in 2027, new types of AI systems will exist that the Act's drafters could not have anticipated.
  • Values in governance: The Act's risk categories reflect specific value judgments about which AI applications are acceptable and which are not. These judgments embed European values about fundamental rights, human dignity, and the precautionary principle.
  • The gap between law and practice: A law on paper is only as good as its implementation. The hard questions raised by each scenario — about ambiguity, enforcement, and interpretation — will be resolved not by the Act's text but by regulators, courts, and the companies and civil society organizations that engage with the regulatory process.