I intuitively sense scope for a meld of such intelligence with the physical world.
Are there startups that are building anything cool in this space?
Regarding the physical world, that's a deeper question. You have people that say LLM's "understand", that they are "intelligent" and that this is an "emergent behavior" of all their weights. You also have people that say they are nothing more than a stochastic parrot or auto-complete on steroids.
I'm in neither camp but let's do a thought exercise. Multi-modal LLM's are training on text, video, and sound. They can know what a chair looks like, what sound it make if you drag it over a wooden floor, and what it would look like when you do that (from this mysterious PoV somewhere). Now take that "knowledge" and ask it to give you 3D coordinates to move a chair right now in the room you're standing in: it simply can't. It's lacking a lot of information about the actual measurements of the room, its own movement capabilities (or those of the human to carry out the task), etc.
There are AI that can do this, but they aren't good for text. We have self-driving cars and factory robots doing things constrained to those domains.
If you say "meld" as in "let's combine a bunch of different AI technologies together with each one doing what it does best", I'm sure people are working on this already. But LLM's are but a small part of solving that problem.
EDIT: if you still can, please add "Ask HN: " to your title here.
Yes, I had moving-the-chair-in-physical-space class of capabilities in mind: robots guided by multimodal intelligence, cars 'surprising me' on a day I'm idle, etc. The challenge here may be in what can be achieved at the edge, the feedback control system for correction of successive prompts.