Most self-driving car simulators are built the hard way, stitched together from whatever the vehicles themselves have already seen and recorded on real roads. Waymo's newest tool takes a different approach entirely: a generative system called the Waymo World Model that can conjure up entire photorealistic 3D driving scenes, including situations its cars have never actually experienced, and use them to train and stress-test its autonomous driving software. Built on top of Genie 3, an advanced general-purpose world model from Google DeepMind, the system marks one of the more ambitious uses yet of generative AI in the self-driving industry, and it's already being used internally to sharpen how the company's vehicles handle the unexpected.

Genie 3 itself is designed to generate immersive, interactive 3D environments from little more than a prompt, and Waymo has adapted that underlying technology specifically for the driving domain. Rather than producing a single flat video feed, the Waymo World Model outputs high-fidelity, multi-sensor data that mimics what the company's actual vehicles perceive on the road, including both camera imagery and lidar point clouds. That distinction matters, since Waymo's Driver system relies on fusing multiple sensor types together in real time, so a simulator that only mimics one of them would only be teaching half the lesson.

What makes the system particularly useful, according to Waymo, is that it isn't limited to replaying scenarios the company's fleet has physically encountered. Because Genie 3 was pre-trained on a massive and highly varied library of video, it carries a kind of general “world knowledge” that lets it generate scenes far outside Waymo's own driving history, from an elephant wandering into the road to extreme weather, natural disasters, or rare, safety-critical near-misses. Engineers can shape those scenes a few different ways — controlling how the simulated vehicle responds to specific driving inputs, adjusting details like road layout and traffic behavior, or simply describing changes in plain language, such as shifting the time of day or generating an entirely synthetic scene from scratch.

The system has one more trick that's arguably more practical than any of the exotic scenarios: it can take an ordinary video clip, including footage from a standard dashcam, and transform it into a full multimodal simulation. That lets Waymo effectively re-create specific real-world locations inside its testing environment and watch how its driving system would behave on that exact stretch of road, grounded in footage that's already tied to reality rather than purely synthetic.

For now, the Waymo World Model remains an internal simulation and training tool rather than something riders will ever directly interact with. Still, Waymo describes the system as a way to build a more rigorous safety benchmark overall, letting its vehicles face down complex, unpredictable situations inside a simulated world first, so they're better prepared for the messier real version when it eventually shows up on an actual street.