Hacker News

550

The Waymo World Model

by xnx1770394842341 comments
Suddenly all this focus on world models by Deep mind starts to make sense. I've never really thought of Waymo as a robot in the same way as e.g. a Boston Dynamics humanoid, but of course it is a robot of sorts.

Google/Alphabet are so vertically integrated for AI when you think about it. Compare what they're doing - their own power generation , their own silicon, their own data centers, search Gmail YouTube Gemini workspace wallet, billions and billions of Android and Chromebook users, their ads everywhere, their browser everywhere, waymo, probably buy back Boston dynamics soon enough (they're recently partnered together), fusion research, drugs discovery.... and then look at ChatGPT's chatbot or grok's porn. Pales in comparison.

by mattlondon1770401371
By leveraging Genie’s immense world knowledge, it can simulate exceedingly rare events—from a tornado to a casual encounter with an elephant—that are almost impossible to capture at scale in reality. The model’s architecture offers high controllability, allowing our engineers to modify simulations with simple language prompts, driving inputs, and scene layouts. Notably, the Waymo World Model generates high-fidelity, multi-sensor outputs that include both camera and lidar data.

How do you know the generated outputs are correct? Especially for unusual circumstances?

Say the scenario is a patch of road is densely covered with 5 mm ball bearings. I'm sure the model will happily spit out numbers, but are they reasonable? How do we know they are reasonable? Even if the prediction is ok, how do we fundamentally know that the prediction for 4 mm ball bearings won't be completely wrong?

There seems to be a lot of critical information missing.

by yummypaint1770415408
> The Waymo World Model can convert those kinds of videos, or any taken with a regular camera, into a multimodal simulation—showing how the Waymo Driver would see that exact scene.

Subtle brag that Waymo could drive in camera-only mode if they chose to. They've stated as much previously, but that doesn't seem widely known.

by xnx1770395429
All this work is impressive, but I'd rather have better trains
by caycep1770411372
Have been seeing Waymo test vehicles regularly around central London recently, operating at speed.

For shits and giggles, I did stop randomly while crossing the road and acted like a jerk.

The Waymo did, in fact, stop.

Kudos, Waymo

by LightBug11770416191
It is great being able to generate a much larger universe of possibilities than what they can gather from real world data collection, but I'd be curious to learn how they check that the generated data is a superset of the possibility-space seen in the real world (e.g. confirm that their models closely match what is seen in the real world too)
by joshfee1770415519
The novel aspect here seems to be 3D LiDAR output from 2D video using post-training. As far as I'm aware, no other video world models can do this.

IMO, access to DeepMind and Google infra is a hugely understated advantage Waymo has that no other competitor can replicate.

by ra71770397494
It’s impressive to see simulation training for floods, tornadoes, and wildfires. But it’s also kind of baffling that a city full of Waymos all seemed to fail simultaneously in San Francisco when the power went out on Dec 22.

A power outage feels like a baseline scenario—orders of magnitude more common than the disasters in this demo. If the system can’t degrade gracefully when traffic lights go dark, what exactly is all that simulation buying us?

by joshuamerrill1770406956
I'd like to see Waymo have a few of their Drivers do some sim racing training and then compete in some live events. It wouldn't matter much to me if they were fast at all, I'd like to see them go into the rookie classes in various games and see how they avoid crashes from inexperienced players. I believe that it would be the ultimate "shitty drivers vs. AI" test.
by ok_dad1770409502
cue the bell curve meme for learning autonomy:

                 ____.----.____
          ______/              \______
    _____/                            \_____
    ________________________________________

    (simulations)  (real world data)  (simulations)
Seems like it, no?

We started with physics-based simulators for training policies. Then put them in the real world using modular perception/prediction/planning systems. Once enough data was collected, we went back to making simulators. This time, they're physics "informed" deep learning models.

by hazrmard1770402349
Deepmind's Project Genie under the hood (pun intended). Deepmind & Waymo both Alphabet(Google) subsidiaries obv.

https://deepmind.google/blog/genie-3-a-new-frontier-for-worl...

Discussed here,eg.

Genie 3: A new frontier for world models (1510 points, 497 comments)

https://news.ycombinator.com/item?id=44798166

Project Genie: Experimenting with infinite, interactive worlds (673 points, 371 comments)

https://news.ycombinator.com/item?id=46812933

by mellosouls1770399870
IIUC, there's a confusion of meaning for "World Model", between Waymo/Deepmind's which is something that can create a consistent world (for use to train Waymo's Driver), vs Yann LeCun/Advanced Machine Intelligence (AMI) which is something that can understand a world.
by AceJohnny21770406823
1. Still hard not to think that this is a huge waste of time as opposed to something that's a little more like a public transport train-ish thing, i.e. integrate with established infrastructure.

2. No seriously, is the filipino driver thing confirmed? It really feels like they're trying to bury that.

by jrm41770405012
Interesting, but it feels like it's going to cope very poorly with actually safety-critical situations. Having a world model that's trained on successful driving data feels like it's going to "launder" a lot of implicit assumptions that would cause a car to get into a crash in real life (e.g. there's probably no examples in the training data where the car is behind a stopped car, and the driver pulls over to another lane and another car comes from behind and crashes into the driver because it didn't check its blindspot). These types of subtle biases are going to make AI-simulated world models a poor fit for training safety systems where failure cannot be represented in the training data, since they basically give models "free reign" to do anything that couldn't be represented in world model training.
by nightpool1770408011
Finally I understand the use case for Genie 3. All the talk about "you can make any videogame or movie" seems to have been pure distraction from real uses like this: limited, time-boxed simulated footage.
by phailhaus1770406213
I wonder if they can simulate the Beatles crossing the street at Abbey Road in the late '60s
by NullHypothesist1770397298
Very impressive work from Waymo. The driving with a tornado in the horizon example kind of struck my imagination, many people actually panic in such scenarios. I wonder though the compute requirements to run these simulations and producing so many data points.
by fabmilo1770406406
It doesn't look like they're going to open sources or anything, but I could imagine this would be great for city planning.

Or the most realistic game of SimCity you could imagine.

by 9999000009991770401067
and I literally just saw the other headline "Waymo says its robotaxis get help from remote workers in the Philippines"
by KeyBoardG1770412842
Dumb question - Why would Waymo disclose this much information to public and competitors?
by pcurve1770410761
Seems relevant: Waymo exec admits remote operators in Philippines help guide US Robotaxis

https://news.ycombinator.com/item?id=46918043

by anigbrowl1770412265
Interesting that this should come out right as lawmakers are beginning to understand that Waymos have overseas operators making major decisions.

[*] https://futurism.com/advanced-transport/waymos-controlled-wo...

by Kapura1770399875
Could these world models be used to build some sort of endless GranTurismo type street racing game?
by t1234s1770407932
This is cool, but they are still not going about it the right way.

Its much easier to build everything into the compressed latent space of physical objects and how they move, and operate from there.

Everyone jumped on the end-2-end bandwagon, which then locks you into the input to your driving model being vision, which means that you have to have things like genie to generate vision data, which is wasteful.

by ActorNightly1770405014
Still needs to be trained on the final boss: dense cities with narrow streets.
by mgaunard1770397154
What if we put this mechanism of recording the world on people. We have mics listening to people talking to us and noises we hear.

Also we record body position actuation and self speech. As output then we put this on thousands of people to get as much data as Waymo gets.

I mean that’s what we need to imitate agi right? I guess the only thing missing is the memory mechanism. We train everything as if it’s an input and output function without accounting for memory.

by threethirtytwo1770411614
Instructions to load it on WAYMAX simulator?
by LowLevelKernel1770404791
Imagine driving in a Waymo 'out of a raging fire'.

Talk about edge cases.

But, what would you do? Trust the Waymo, or get out (or never get in) at the first sign of trouble?

by PeterStuer1770398025
What is the 5/3 tiles? Cameras?
by ge961770404327
For whatever it’s worth World models is going to be the dominant computing structure of the future

I started working heavily on realizing them in 2016 and it is unquestionably (finally) the future of AI

by AndrewKemendo1770403310
Nvidia has had this for years. What am I missing?
by 011000111770404340
This page crashes my browser.

Vivaldi 7.8.3931.63 on iOS 26.2.1 iPhone 16 pro

by tgrowazay1770402632
One interesting thing from this paper is how big of a LiDaR shadow there is around the waymo car which suggests they rely on cameras for anything close (maybe they have radar too?). Seems LiDaR is only useful for distant objects.
by mempko1770400154
This might be relevant to the timing here: https://eletric-vehicles.com/waymo/waymo-exec-admits-remote-...
by jimt12341770398176
The new frontier is manifestly the Phillipines.
by Vosporos1770401627
Seems interesting, but why is it broken. Waymo repeatedly directed multiple automated vehicles into the private alley off of 5th near Brannan in SF even after being told none of them have any business there ever, period. If they can sense the weather and stuff then maybe they could put out a virtual sign or fence that notes what appears to be a road is neither a through way nor open to the public? I'm really bullish on automated driving long term, but now that vehicles are present for real we need to start to think about potentially getting serious about finding some way to get them to comply with the same laws that limit what people can do.
by m0llusk1770398438
[dead]
by selenajennifer1770414402
[dead]
by andrewmcwatters1770399659
[flagged]
by xvxvx1770406428
Wow, interesting timing for this PR blast considering the admission in the Senate Commerce Committee hearing. Not transparent at all!
by devmor1770397465
How many Filipinos, who do not have US drivers licenses, does it take to drive this new model?
by turtlesdown111770399349
The Waymo driving model: hire some guys in Philippines: https://futurism.com/advanced-transport/waymos-controlled-wo...
by smcl1770404297
What's going to happen to all the millions of drivers who will lose their job overnight? In a country with 100 million guns, are we really sure we've thought this through?
by OGEnthusiast1770396562