2026-06-18 — views

AV Pedestrian and Cyclist Detection — The Hardest Perception Problem and Safety Data

Pedestrians and cyclists are the hardest targets for AV sensors — small, fast, unpredictable. Here is what the detection science and safety data show.

Article 59 in the Physical AI Benchmark Series — The Hardest Perception Problem

Of all the objects an autonomous vehicle must detect, pedestrians and cyclists are the most consequential and the most technically difficult. They are the road users who die when something goes wrong — and they are the ones that sensors struggle with most. A car is a large, rigid, radar-reflective box moving on a predictable trajectory. A pedestrian is a small, articulated, low-radar-cross-section object that can change direction instantaneously, emerge from nowhere, and appear in an almost infinite variety of appearances. A cyclist is faster, more maneuverable, and travels in a space between vehicle and pedestrian that road infrastructure rarely designs for clearly.

This article maps the specific detection challenges, what each sensor modality contributes, where current production systems differ in their approaches, and what the available safety data shows.

Section 1 — Why Pedestrians and Cyclists Are Uniquely Hard

The difficulty is not any single factor but the compounding of many. Each challenge below adds to the others.

Challenge	Detail
Small size	A pedestrian occupies roughly 0.5 m² of frontal cross-section versus a car’s roughly 6 m². Radar cross-section is much smaller still — a pedestrian returns a weak radar signal that overlaps with noise from other small objects.
Unpredictable motion	Pedestrians can change direction instantly with no signal; children especially exhibit sudden lateral movements. A car’s trajectory can be predicted with reasonable confidence over a 2–3 second horizon; a pedestrian’s cannot.
Occlusion and sudden appearance	Pedestrians emerge from between parked cars, around building corners, from bus doors — appearing in the sensor field with zero warning time. There is no equivalent for vehicles, which approach from established lanes.
Articulated body	Arms and legs move independently from the torso. Detecting a pedestrian is not finding a rigid bounding box but recognizing a deformable object whose limbs have separate motion vectors. Pose estimation is required to understand gait and intent.
Appearance variation	A pedestrian might be wearing a bright yellow jacket or a dark coat at night, carrying an umbrella, pushing a stroller, in a wheelchair, or wearing a costume. The visual diversity is orders of magnitude larger than vehicle appearance variation.
Low-light vulnerability	More than 75% of pedestrian fatalities in the US occur in darkness (NHTSA data). Human drivers are degraded at night; camera-based AV systems face the same degradation without headlight illumination as a reference.
Group dynamics	Crowds at crosswalks, multiple pedestrians in close proximity with partially occluding each other — multi-agent prediction of group behavior is significantly harder than tracking individual objects.
Edge cases	Wheelchair users, people with unusual gait from injury or disability, police officers directing traffic with non-standard gestures, children on play equipment near roads, costumed characters at events — the long tail of appearance and behavior variation is very long.

Cyclists add speed to this complexity. A cyclist travels at 15–25 mph, faster than a pedestrian but sharing road space with pedestrians at intersections and crossings. Hand signals are small and brief. Lane position is often ambiguous. And like pedestrians, cyclists appear in highly variable configurations: with cargo bags, trailers, helmets, without helmets, in groups, alone.

Section 2 — How Each Sensor Handles Pedestrians

No single sensor solves pedestrian detection. The practical question is which combinations provide the best coverage across the failure modes.

Sensor	Pedestrian detection strength	Key limitation
Camera (visible light)	Excellent in daylight conditions: color, texture, and body pose are all captured; deep-learning detectors (YOLO-family, DETR-based architectures) achieve high accuracy in standard well-lit conditions. Video (temporal sequences) enables motion-based cues that single frames miss.	Night: performance degrades significantly without sufficient illumination. Heavy rain: contrast loss and lens droplets degrade image quality. Occlusion: cannot see through solid objects; partial body detection requires inference.
LIDAR	Generates a 3D point cloud that is largely independent of lighting conditions. Can detect a pedestrian’s leg emerging from behind a parked car before the full body is visible — a key advantage for occlusion scenarios. 3D bounding boxes enable range estimation independent of appearance.	Very low-reflectivity clothing (dark winter coats) reduces return intensity. Heavy rain attenuates the laser beam. Small objects at long range return fewer points, reducing confidence at distance.
Radar	Detects movement and radial velocity (Doppler) reliably through rain and fog. Robust in adverse weather where LIDAR and cameras degrade.	Low angular resolution — cannot distinguish a pedestrian from a small animal, trash can, or mailbox by shape. Provides velocity and approximate range but no shape or pose information. False positives from roadside infrastructure are common.
Thermal infrared (IR)	Detects body heat directly; works in complete darkness without any ambient or artificial illumination. Pedestrians appear as warm objects even in zero-light environments.	Expensive sensor; limited availability in production vehicles. Limited resolution compared to visible cameras. Does not provide shape or pose detail; classification is harder. Environmental heat sources (hot pavement, vehicle engines) can create clutter.
Sensor fusion	LIDAR provides 3D position and shape; camera provides appearance classification and pose; radar provides velocity and weather-robustness. Combined, the system can detect, classify, track, and predict pedestrian intent with higher confidence than any single sensor.	Fusion complexity introduces its own failure modes. If the fusion algorithm incorrectly merges detections from different sensors, it can create false negatives (dismissing a real detection) that are harder to catch than individual sensor errors.

The critical insight about fusion: it is not simply additive. A LIDAR that detects a pedestrian-shaped object, a camera that classifies it as human, and radar that confirms it is moving toward the road — all three confirming the same object increases confidence enormously. But if the LIDAR detection misaligns with the camera frame due to a calibration drift, the fusion might discard the detection rather than flag it. Maintaining sensor calibration in production vehicles over years of road vibration is an underappreciated operational challenge.

Section 3 — Tesla’s Camera-Only Pedestrian Detection

Tesla’s Full Self-Driving system is built around a camera-first philosophy, with no LIDAR or radar in current production FSD vehicles (radar was removed from most models starting 2021). Pedestrian detection therefore relies entirely on neural network inference from camera images.

Dimension	Detail
Detection architecture	FSD uses an end-to-end neural network approach (v12 architecture) trained on a very large fleet-collected dataset. The system processes video sequences (not single frames), enabling temporal context for occlusion handling.
Scale advantage	The Tesla fleet has collected an enormous volume of diverse pedestrian encounters across geographies, weather conditions, and time of day. The training dataset scale is a genuine competitive advantage for covering the appearance variation challenge.
Daylight performance	In daylight, standard urban pedestrian detection (people crossing at crosswalks, pedestrians on sidewalks, cyclists in bike lanes) performs well. The deep learning system can distinguish pedestrians from poles, dogs, trash cans, and other similarly sized objects.
Night weakness	Without LIDAR, the system depends entirely on what headlights illuminate and ambient light. A pedestrian in dark clothing on a poorly lit road receives minimal illumination from headlights at relevant stopping distances. This is the camera-only system’s most significant vulnerability for pedestrian safety.
Temporal occlusion handling	If a pedestrian was visible in frames two seconds ago and has now been occluded, the model maintains an inferred trajectory of where that pedestrian likely is. This is a meaningful capability but is an inference, not a measurement.
Intent prediction	FSD v13 improved at reading pedestrian intent signals — head turn direction, body lean toward the road, raised hand at a crosswalk. These are real behavioral cues that human drivers use, and teaching them to a neural network is meaningful progress. Performance is still imperfect and not independently verified (est.).
Phantom braking history	Earlier FSD versions had elevated phantom braking from false pedestrian detections (mistaking shadows, plastic bags, or bushes for pedestrians). FSD v12 and v13 significantly reduced this problem, reflecting the value of fleet-scale training data for reducing false positives.
Driverless safety data	As of mid-2026, Tesla FSD operates under human supervision — the human driver is expected to monitor and intervene. No equivalent driverless pedestrian interaction safety database at the scale of Waymo’s published robotaxi data exists.

The camera-only architecture is a coherent design choice with a clear logic: it is cheaper, the sensor is more mature, and the intelligence is scalable through software and training data. The debate is whether software and data can compensate for the physical absence of LIDAR in low-light scenarios where a 3D sensor would have provided range information independent of illumination.

Section 4 — Waymo’s Multi-Sensor Pedestrian Detection

Waymo equips its vehicles with a suite of sensors specifically designed so that no single sensor failure creates a detection blind spot. For pedestrians, this means LIDAR is the primary detection sensor and camera provides confirmation and classification detail.

Dimension	Detail
LIDAR primary role	The 3D point cloud detects pedestrian shape — even at night, even in rain, without illumination from headlights. A pedestrian walking in complete darkness at 50 m range returns a point cluster that the LIDAR classifier identifies as human-shaped. Night and day performance are substantially equal.
Camera confirmation	Camera adds color, texture, clothing detail, and body pose estimation to the LIDAR-detected object. This enables finer classification (adult vs child, cyclist with cargo vs without) and intent inference from pose.
Radar velocity layer	Radar confirms the detected object is moving and provides a velocity vector. This helps distinguish a stationary pedestrian standing on a sidewalk from a pedestrian about to step into the road — combined with LIDAR position, the system can track both trajectory and speed.
Occlusion advantage	LIDAR detects the legs of a pedestrian emerging from behind a parked car before the full body is visible. At 30 m range, this provides an additional 0.5–1.0 second of warning compared to a camera system that requires the full body to be visible. At city driving speeds, that margin is meaningful.
Range in darkness	LIDAR detects a pedestrian at 50–80 m range even in complete darkness (est.). A camera system relying on headlight illumination sees roughly 40 m ahead with standard low beams at highway-adjacent speeds — a gap that matters at intersections with poor street lighting.
Published safety data	Waymo’s 2023 safety report covering approximately 7 million driverless miles reported zero serious pedestrian injuries attributable to Waymo system fault (from published data). This is a directional finding, not a definitive statistical comparison — the operating environment (primarily urban San Francisco and Phoenix) and definition of “serious injury” differ from the NHTSA baseline.
Cyclist-specific detection	Cyclists move faster than pedestrians (15–25 mph), making trajectory prediction more time-sensitive. LIDAR tracks the bicycle frame and rider as a combined object. Camera classifies hand signals and body position. Radar provides velocity confirmation. The multi-sensor stack enables earlier confident classification than camera alone (est.).

The Waymo sensor architecture reflects the belief that the failure mode for pedestrian detection is unacceptable — that the consequence of a missed pedestrian is severe enough to justify the cost and complexity of LIDAR alongside cameras. The cost of this approach is exactly that: cost and complexity, plus the size and power draw of a full sensor suite.

Section 5 — The Safety Comparison: AVs vs Human Drivers

The central question for AV investment and regulation is whether the technology is demonstrably safer than the human baseline for the road users who are most at risk. The honest answer as of mid-2026 is: directionally yes for LIDAR-equipped robotaxis in their operational domains, but the data is not yet at sufficient scale for statistically definitive conclusions.

Metric	Human drivers (NHTSA baseline)	Waymo (published, 2023)	Tesla FSD (supervised)
Pedestrian fatalities per 100M miles	Approximately 1.75 (NHTSA US average, recent years)	Zero serious pedestrian injuries in approximately 7M driverless miles (not directly comparable to NHTSA rate — different operating domain and denominator)	No driverless data; supervised disengagement rate is the available proxy metric
Night pedestrian risk	Human driver risk roughly 3x higher at night vs daytime (consistent with the 75% night fatality statistic)	LIDAR-equipped systems: no meaningful night/day performance difference	Camera-only: night performance is materially harder (est.); quantification requires independent testing
Jaywalking pedestrian	Human driver reacts to visible pedestrian; reaction time 0.7–1.5 s from detection	Waymo models pedestrian crossing as a probability distribution; LIDAR detects lateral movement toward road before visible to a camera at the same range	FSD neural net predicts intent from body pose and head direction; capability confirmed in v13 changelog but not independently benchmarked
Impairment	Approximately 25% of fatal crashes involve impaired drivers (NHTSA); 100% of pedestrian fatalities have a human or AV driver in the vehicle	Never impaired	Never impaired
Distraction	Cell phone distraction is a factor in approximately 9% of fatal crashes (NHTSA)	Never distracted	Never distracted

On the comparison methodology: The 7 million driverless Waymo miles and the NHTSA national baseline are not directly comparable. Waymo operates primarily in urban areas of Phoenix and San Francisco — higher pedestrian density than the US average (which includes vast rural miles), but also lower highway speeds where kinetic energy in a crash is lower. The Waymo fleet has not operated in rural highways, snowstorms, or the many edge cases the national fleet encounters. The directional signal from the published data is positive, but the caveat applies: it is early data from a carefully selected operating domain.

What the data does not yet show — and what is needed to make a definitive case — is millions of driverless miles across diverse weather, geography, and pedestrian density conditions, with a consistent definition of serious injury matched to NHTSA standards. That data will be available as Waymo continues to expand its operational domain. Until then, the honest characterization is: current AV robotaxis in their operational domain appear to be performing better than the human baseline for pedestrian safety, but the data is too narrow to generalize.

The camera-only vs LIDAR debate matters most here. If night pedestrian detection is the single highest-risk scenario (consistent with the NHTSA night fatality statistic), the sensor architecture choice is a direct safety architecture choice. How that resolves in practice — whether neural network scaling with camera data can close the gap, or whether physical sensor diversity is required — is one of the most consequential technical questions in the field.

Sources: NHTSA Fatality Analysis Reporting System (FARS) — nhtsa.gov; Waymo Safety Report 2023 — waymo.com/safety; Tesla Vehicle Safety Report — tesla.com/VehicleSafetyReport; IEEE Transactions on Intelligent Transportation Systems — ieeexplore.ieee.org. All figures marked (est.) are estimates derived from public company materials, industry reporting, and analyst research. They have not been independently verified and should be treated as directional. This article does not constitute investment advice.