No you don’t. I can feed images of crazy driving scenarios into LLaVA and get reasonable responses. That’s a general purpose LLM with $500 worth of fine tuning running locally on my PC. You should look into what can be done with the current state of the art LLMs. Your intuition for what’s possible is out of date.

If I can do that with open source LLaMA variants, I can only imagine what’s possible if you have an actual annotated dataset of driving scenarios. Imagine a LLaMA model thats been fine tuned for lane selection, AEB, etc.