Case Study 1: The Self-Driving Car That Couldn't See

Autonomous Vehicle Vision Failures and What They Teach Us

The Promise

Few technologies capture the imagination like self-driving cars. The promise is extraordinary: roads without drunk drivers, mobility for people who cannot drive, optimized traffic flow, and a dramatic reduction in the roughly 40,000 annual traffic fatalities in the United States alone. At the heart of every autonomous vehicle is a computer vision system — the "eyes" that allow the car to perceive the road, other vehicles, pedestrians, traffic signs, lane markings, and obstacles. If the car can't see reliably, nothing else matters.

By the mid-2010s, the technology seemed to be progressing faster than anyone expected. Autonomous vehicles were logging millions of test miles. Industry leaders predicted fully self-driving cars would be commercially available by 2020. Investment poured in — tens of billions of dollars. Computer vision, combined with lidar, radar, and sensor fusion, appeared ready to replace the human eye behind the wheel.

Then reality set in.

The Incident: Tempe, Arizona, March 2018

On the evening of March 18, 2018, Elaine Herzberg was walking her bicycle across a four-lane road in Tempe, Arizona. An Uber self-driving test vehicle, a modified Volvo XC90 SUV traveling at approximately 39 miles per hour, struck and killed her. A safety driver was behind the wheel but was not watching the road at the time of the collision.

The National Transportation Safety Board (NTSB) investigation revealed a cascade of failures in the vehicle's perception system — failures that illuminate the fundamental limitations of computer vision in the real world.

What the system saw: The car's sensors — a combination of radar, lidar, and cameras — detected Herzberg approximately 5.6 seconds before impact. That should have been enough time to brake. But the perception system couldn't figure out what she was. Over those critical seconds, the system's classification oscillated:

• First, it classified her as an "unknown object"
• Then as a "vehicle"
• Then as "other" — not matching any trained category
• Then as a "bicycle"
• Then as a "vehicle" again

Each time the classification changed, the system's prediction of her path reset. It never sustained a classification long enough to generate a coherent prediction of where she was headed. The emergency braking system, which had been disabled to prevent erratic behavior during testing, never engaged.

Why it failed: Herzberg was walking a bicycle across a road at night, outside of a crosswalk. This scenario — a pedestrian walking alongside (not riding) a bicycle, in a dark area, crossing outside of designated zones — was an edge case the system had not been adequately trained to handle. The categories it had learned (pedestrian, vehicle, bicycle, unknown) couldn't accommodate the ambiguity of a person merging with a bicycle-shaped object in low light.

The Deeper Problem: Edge Cases at Scale

The Tempe crash was a tragedy, but it was not the only incident revealing the limits of autonomous vehicle vision. A pattern of failures has emerged across the industry:

Tesla Autopilot and emergency vehicles: Between 2018 and 2023, Tesla vehicles operating with Autopilot were involved in over a dozen documented collisions with stationary emergency vehicles — fire trucks, police cars, and ambulances — parked on the side of highways with flashing lights. The vision system, trained primarily on flowing traffic, did not reliably recognize that a stationary vehicle in a travel lane was an obstacle to avoid rather than a parked car on the shoulder.

The white truck problem: In 2016, a Tesla Model S on Autopilot drove underneath a tractor-trailer that was crossing the highway, killing the driver. The system's vision failed to distinguish the white side of the trailer from the bright sky behind it. Humans rarely make this mistake — our depth perception, contextual understanding, and knowledge of physics tell us that a large flat surface crossing our path is a solid obstacle. The camera-based system lacked these cues.

Weather limitations: Autonomous vehicles have consistently struggled with conditions that degrade visual information: heavy rain, fog, snow covering lane markings, low sun causing glare, and puddles creating reflections that confuse lane-detection algorithms. These are conditions that human drivers also find challenging — but human drivers compensate with experience, caution, and common sense. They slow down. They follow the car ahead. They recognize that foggy conditions call for extra vigilance. Current vision systems lack this adaptive reasoning.

What the Failures Reveal

These incidents are not evidence that autonomous driving is impossible. They are evidence that computer vision — even when paired with lidar, radar, and sophisticated sensor fusion — does not "see" the way humans see.

The category problem: Computer vision systems classify the world into categories they learned during training. An object that doesn't fit neatly into a learned category — like a pedestrian pushing a bicycle — creates dangerous uncertainty. Human vision doesn't rely on categorical classification in the same way. We perceive objects as physical things with mass, trajectory, and causal properties, whether or not we can name them.

The common-sense problem: A child chasing a ball into the street is visually similar to a child standing on the sidewalk near a ball. A human driver understands the danger of the first scenario because we understand what children do and what balls do. A vision system sees pixel patterns. Infusing computer vision with this kind of causal, common-sense reasoning remains one of the great unsolved problems in AI.

The long tail problem: Self-driving systems perform well on the scenarios that dominate their training data: clear weather, standard road markings, typical vehicles at typical speeds. But the real world has a "long tail" of unusual events — a deer on the highway, a child's Halloween costume that doesn't look like a standard pedestrian, a fallen tree, a traffic cop using hand signals that override signal lights. Each unusual event is individually rare, but collectively they are common. You encounter something unusual almost every time you drive.

The Human-in-the-Loop Question

The Tempe crash highlighted another dimension: the human safety driver. Uber had placed a human behind the wheel as a backup, but the NTSB investigation found that the safety driver was watching a video on her phone at the time of the crash. This is a predictable failure of the "human in the loop" model: when humans are asked to monitor a system that usually works, they disengage. Attention is a finite resource, and monitoring a machine that handles 99% of situations correctly is extraordinarily boring.

This creates a paradox. The better the autonomous system gets, the less attentive the human backup becomes. But the moments when the human needs to intervene — the remaining 1% of edge cases — are precisely the most difficult driving scenarios. The system fails at the worst possible time, and the human isn't ready to take over.

This dynamic isn't unique to driving. It appears wherever humans supervise AI systems: in aviation (automation complacency), in radiology (over-reliance on AI flagging), and in content moderation (trusting the system's initial filter). Chapter 8 will explore these failures of human-AI collaboration in more depth.

Where the Industry Stands

As of the mid-2020s, the autonomous vehicle industry has evolved significantly from its early hype:

• Waymo (Alphabet/Google) operates commercial robotaxi services in Phoenix, San Francisco, and Los Angeles, but in carefully mapped, geofenced areas with favorable weather conditions. The company uses a combination of lidar, cameras, and radar, and has logged millions of miles of autonomous driving.
• Cruise (General Motors) expanded and then paused operations after an incident in San Francisco in October 2023 in which a pedestrian was dragged by one of its vehicles.
• Tesla continues to pursue a camera-only approach with its "Full Self-Driving" system, which despite its name requires constant driver supervision and is classified as a Level 2 driver-assistance system, not autonomous driving.
• Industry consensus has shifted from "fully autonomous vehicles will be everywhere by 2020" to "autonomous vehicles will be deployed gradually, in specific conditions, with ongoing human oversight."

The gap between demonstration and deployment — between showing that something works in favorable conditions and ensuring it works in all conditions — turns out to be enormous. And computer vision is at the center of that gap.

Discussion Questions

1. The Tempe crash involved an edge case: a pedestrian walking a bicycle across an unlit road at night. Should autonomous vehicles be required to handle every possible edge case before deployment, or is some threshold of overall safety sufficient? Who should set that threshold?

2. If autonomous vehicles are statistically safer than human drivers overall but fail in different ways (edge cases rather than inattention, intoxication, or fatigue), how should society evaluate the trade-off? Is it acceptable for an AI to cause different kinds of accidents even if it causes fewer total accidents?

3. The safety driver in the Tempe crash was watching a video. Given what we know about human attention and automation complacency, was the "human in the loop" model for autonomous vehicle testing fundamentally flawed? What alternatives can you imagine?

4. Autonomous vehicle companies test primarily in cities with favorable conditions — sunny weather, well-maintained roads, clear lane markings. What are the equity implications of deploying autonomous vehicles only in these environments?

5. Connect this case to the MedAssist AI example from the chapter. Both involve computer vision systems that work well in the scenarios represented in their training data but fail for underrepresented scenarios. What does this pattern tell us about how computer vision systems should be evaluated before deployment?