In Chapters 5 and 6, we looked at traditional robotics pipelines where systems are divided into explicit stages: sensing, object detection, path planning, and motor control.
Now, we explore a radically different philosophy: End-to-End Autonomous Driving. Instead of writing hand-crafted rules or breaking driving into separate modules, can a machine learning model learn to drive directly by observing a human expert? In this chapter, we investigate behavioral cloning, pixel-to-action mappings, data collection strategies, and dataset bias.
In traditional modular robotics, control follows a strict pipeline: a camera feeds an object detector, which feeds a path planner, which feeds a motor controller, which finally drives the wheels. If any intermediate module fails (e.g., if object detection misses a sign), the whole system breaks down.
End-to-End Learning replaces these intermediate steps with a single machine learning model (often a Deep Neural Network). The model takes raw camera/vision inputs and directly outputs motor control commands: camera and vision inputs feed straight into the end-to-end AI model, which outputs motor commands \((v, \omega)\) directly—no intermediate object detector or path planner in between.
Training: An AI model is trained on these recorded pairs until it learns to output the same steering commands as the human when presented with similar visual scenes.
During the data collection phase, the human driver’s joystick commands \((v, \omega)\) and the onboard camera stream are recorded together in lockstep, producing a dataset of \((\text{Image}, v, \omega)\) triples that the model later learns from.
NVIDIA PilotNet: One of the pioneering end-to-end driving models. A car equipped with three front-facing cameras drove thousands of miles driven by humans. The neural network learned lane-keeping and road curvature steering directly from raw steering wheel angle recordings without ever being programmed with explicit lane-finding algorithms.
Wayve AI: Modern autonomous delivery vans use end-to-end AI to navigate busy urban streets, learning complex maneuvers like roundabouts and pedestrian yield responses through imitation and reinforcement learning.
Subsubsection7.1.2.2VEX AI Competition (VAIC) Applications
In high-level VEX AI matches, time during autonomous periods is critical. Rather than writing long sequences of driveFor() and turnFor() commands with manual conditional branches:
Direct Sensor Mappings: The vision system pairs the VEX AI Vision camera features (e.g., target tracking coordinates) directly with motor velocity outputs \(v\) (forward speed) and \(\omega\) (turning rate). The robot learns to mimic the smooth line choices of the human driver when driving toward scoring goals or game pieces.
Modular pipelines require cameras, whereas end-to-end models do not use any sensors.
Incorrect. End-to-end models still rely on raw camera/vision sensor input; they just skip the intermediate hand-crafted processing stages.
End-to-End models eliminate intermediate hand-crafted stages (like explicit object detection) by mapping sensor inputs directly to motor outputs.
Correct! An end-to-end model replaces the entire chain of object detection, path planning, and motor control with a single learned mapping from raw sensor input to motor commands.
Behavioral cloning only works on 4-wheeled vehicles, while modular pipelines work on all robots.
Incorrect. Behavioral cloning is a general learning approach and is not restricted to any particular wheel configuration.
Modular pipelines use supervised learning, while end-to-end pipelines use unsupervised clustering exclusively.
Incorrect. Behavioral cloning is itself a form of supervised learning, trained on labeled (image, control command) pairs.
