Chapter 6 Key Takeaways: Computer Vision

One-Sentence Summary

Computer vision systems convert images into grids of numbers and learn to extract increasingly complex patterns from those numbers, but they process pixel statistics rather than understanding the visual world — a distinction with profound consequences for accuracy, fairness, and safety.

Core Concepts

  1. Images as numbers. Every digital image is a grid of pixels, each storing numerical color values. Computer vision begins by processing these numbers — it never sees "objects" the way humans do.

  2. Hierarchical feature learning. Convolutional neural networks (CNNs) learn to detect visual patterns in layers: edges in early layers, textures and shapes in middle layers, parts and objects in later layers. This hierarchy is learned from data, not programmed by hand.

  3. Classification, detection, segmentation. Computer vision tasks range from labeling an entire image (classification) to locating objects within it (detection) to labeling every pixel (segmentation). Each level adds precision and complexity.

  4. Facial recognition. FRT detects faces, extracts geometric features into a face embedding, and matches against a database. The technology is powerful but has documented accuracy disparities across race and gender, and raises fundamental questions about consent and civil liberties.

  5. Deepfakes. AI-generated synthetic media can produce realistic but fabricated videos and images. The "liar's dividend" — the ability to dismiss real evidence as fake — may be as damaging as the fakes themselves.

  6. Adversarial examples and edge cases. Vision systems can be fooled by imperceptible pixel-level changes (adversarial examples) and fail on scenarios not represented in training data (edge cases). These failures reveal that CNNs learn pixel statistics, not visual understanding.

Key Terms

Term Definition
Pixel The smallest addressable element of a digital image, storing one or more numerical color values
Feature A visual pattern (edge, texture, shape) that a vision system detects at a particular layer of processing
Convolutional neural network (CNN) A neural network architecture that slides learned filters across images to detect hierarchical patterns
Image classification Assigning a single label to an entire image
Object detection Identifying multiple objects in an image and locating each with a bounding box
Image segmentation Assigning a category label to every pixel in an image
Face embedding A numerical representation of a face's unique geometric features used for recognition and matching
Deepfake Synthetic media generated by AI, typically showing people saying or doing things they never did
Adversarial example An input with carefully calculated, often imperceptible modifications designed to cause a neural network to misclassify it
Edge case A scenario that falls outside the patterns represented in training data, where system performance may degrade

Key Frameworks

Evaluating a facial recognition deployment: 1. Purpose — Is FRT the least invasive solution? 2. Accuracy — Has it been tested across demographics? 3. Consent — Are subjects informed? 4. Data governance — Who stores the data and for how long? 5. Accountability — Who is responsible for errors? 6. Proportionality — Do benefits justify the risks? 7. Alternatives — Could a non-biometric approach work?

Connections to Recurring Themes

  • Tools built by humans: Training data reflects the demographics and conditions of who collected it. MedAssist AI's performance gaps trace directly to which hospitals contributed data.
  • Capability vs. understanding: CNNs classify images with superhuman accuracy on benchmarks but don't understand what they see. Adversarial examples prove this.
  • Who benefits, who is harmed: Facial recognition benefits institutions (law enforcement, corporations) while risks fall disproportionately on individuals, especially people of color.
  • Human in the loop: Whether in radiology, driving, or policing, humans remain essential because machines process pixels, not consequences.

What to Remember for Later Chapters

  • Chapter 7 (AI Decision-Making): The vision pipeline is one input to a larger decision system. Understanding how visual inputs become actionable decisions is the next step.
  • Chapter 8 (When AI Gets It Wrong): The failure modes discussed here — edge cases, adversarial examples, demographic accuracy gaps — are specific instances of broader AI failure patterns.
  • Chapter 9 (Bias and Fairness): The Gender Shades study is a foundational example of AI bias auditing that will be revisited.
  • Chapter 12 (Privacy and Surveillance): Facial recognition is a cornerstone case in the broader discussion of AI-enabled surveillance.