Hacker News Reader

Comment by drakenot | Hacker News Reader

Are LLMs going to eventually become the architecture that powers self-driving cars?

Seeing them play Portal and other video games, I'm curious if they will eventually help solve that last N% of self-driving.

A VLM (a vision-language model) is already being used by Waymo[1]. It's useful for scenarios that require reasoning and general knowledge.

Fast forward to me sitting at a Green Light waiting for my usage to reset for the week so I can get to where I’m going

To my understanding gemma is shipped with waymo, in a highly modified fashion.

I'm skeptical, tho. Cost will push for right sizing, much like we have right sized a lot of things about modern cars.

In a sense, they already are. The giant leap in self driving cars we've seen in the last handful of few comes from using transformer models.

I remain worried about prompt injection style attacks against self-driving cars.

Imagine if someone finds a weird image pattern that gets misinterpreted as instructions and hangs that off a bridge over a freeway.

It'll get rooted over the uplink/WiFi/BT long before that. Probably even more likely for non-SDVs.

GPT-6 is multimodal, LLMs alone have no vision capability

"LLM" is now in practice a superset of "LMM"

It's not a coincidence that Waymo started becoming viable after GPT-3.

News to me. Where did you learn about that?