Hacker News Reader

Comment by nasreddin | Hacker News Reader

Its an incorrect assumption, the inference speed and particularly the inference speed of the on-device LLMs with which AVs would need to be using is not compatible with the structural requirements of driving.

View on HN

I think the assumption is valid. Most of the reasoning components of the next gen (and some current gen) robotics will use VLMs to some extent. Deciding if a temporary construction sign is valid seems to fall under this use case.

But unless you are using a single, end-to-end model for the entire driving stack, that "proceed" command will never influence accelerator pedal.

Sure, there will be a VLM for reading the signs, but the worst it'd be able to output is things like "there is a "detour" sign at (123, 456) pointing to road #987" - and some other, likley non-LLM, mechanism will ensure that following that road is actually safe.

Not a "proceed" command but they can influence the accelerator. I had a dodge ram van that would constantly decelerate on cruise control due to reading road signs. The signs in some states like California for trucks towing trailers are 55 mph but the speed limit would be 65 or 70 mph. The cruise control would detect the sign and suddenly decelerate to 55.

That's an example of things working as expected - the sign recognition system is very limited, in that it can only return road sign information. So it can _ask_ cruise control system to change the speed, but it's up to cruise control to decide if it's safe to obey the request or not. For example, I am pretty sure it'll never raise the speed, no mater what sign recognition system says.

Reply on news.ycombinator.com