Google actually recently went some steps further and combined the PaLM LLM (bigger than GPT-3.5) with a 22 billion parameter Vision Transformer to do this -

https://palm-e.github.io

there were reports from Microsoft recently as well. If I remember correctly their version of ChatGPT given a task in plain English generated actions script for a robot.

So, we are getting closer to AI 'Goblin'. Almost generic, sub-human, embodied AI