An example I saw:
- At the start of a construction area, a guy wearing a hi-viz vest holds a stop sign. A self-driving car stops at the sign.
- The guy _lowers_ the sign a bit while looking over his shoulder down the street towards others on his crew.
At this point a _human_ guesses that the sign is lowered only b/c the guy has seen that the car stopped, and expects the car to stay stopped until some further signal (e.g. a waving gesture, or flipping the sign to show the "slow" side). The human driver understands that stop sign guy is looking to coordinate with someone else nearby. There's a "script" for this kind of interaction.
... but the self-driving car starts moving as soon as the road crew guy lowers the sign. In this case nothing seriously bad happened. But it was not following The Conventions.
This doesn't take full general intelligence perhaps -- but it takes some greater reasoning about what people are doing than the cars seem to have currently, and so sometimes they drive into a zone that the fire department is actively using to fight a fire, and get in the way.
If I can do that with open source LLaMA variants, I can only imagine what’s possible if you have an actual annotated dataset of driving scenarios. Imagine a LLaMA model thats been fine tuned for lane selection, AEB, etc.
LLMs with vision work completely differently. You’re leveraging the world model, built from a terabyte of text data, to aid your classification. The classic example of an image they handle well is a man ironing clothes on the back of a taxi. Where traditional image classifiers wouldn’t have a hope of handling that, vision LLMs describe it with ease.
i.e. radar based deceleration to avoid accidents already helps many humans avoid collision. It'll help the robot too.
The rest of driving is relatively simple, methodological, and slow.
Waymo in Phoenix and current cruise cars in SF seem like good counter examples.
The bar is also a lot lower - human drivers are pretty bad.