end-to-end training
Training an entire system as a single unit through interaction, as opposed to stitching together separately pretrained modules.
As defined in 230 | Raphaël Millière on How Artificial Intelligence Thinks
Where it comes up · 2
Most important claims · 3 of 3Search all claims for “end-to-end training” →
- Projects like Google's SayCan plug pretrained language models into robots to translate natural-language instructions into actionable sub-tasks using a vision-language model, but the key limitation is that these systems are not trained end-to-end through world interaction—they bolt together passively-pretrained modules—raising a deep question about whether such modular stitching, versus learning from the ground up via rich multimodal interaction, is the right model of cognition.230 | Raphaël Millière on How Artificial Intelligence Thinks
- Tesla's end-to-end autopilot training never taught the system what a car, person, or cyclist is, nor how to read—it learned all objects on the road purely from watching video, just like humans, who are fundamentally 'photons in, controls out': the vast majority of information reaching our brain is from our eyes and the output is motor signals, and the same is true of the car.Elon Musk #4
- The work done in the car translates to the robot because both are fundamentally 'video in, controls out' systems—a car is a robot on four wheels and Optimus is a robot with hands and legs—so the same end-to-end learning machinery applies with only a different set of controls, enabling the robot to learn tasks like threading a needle by watching, potentially within a year.Elon Musk #4