Holodeck: Language Guided Generation of 3D Embodied AI Environments | Hacker News Reader

Holodeck: Language Guided Generation of 3D Embodied AI Environments

yueyang1996.github.io

55 pointsgeox7 comments

I recently rewatched the Star Trek TNG episode where Geordi LaForge accidentally creates a self aware Moriarty as an opponent for Data role playing Sherlock Holmes. It struck me how similar the Holodeck prompts resemble prompts to get ChatGPT to tell a story with certain parameters.

Honestly LLMs in general give me that vibe. Especially character description; "Computer, create an assistant for me. Make him enthusiastic and knowledgeable, and always provide well-written, clear, factual responses." Star Trek or Ollama?

I hooked up a small label printer to a custom application an LLM running on a Jetson the other week that now lives in my workshop permanently. "Hey jiji, I need a label for this piece of 3/4 birch. It's the left panel of the upper cabinet for the 3D printer. 14 inches by 23 and a half inches."

And the LLM spits out the parameters that invoke a local function to print the label which I can stick on the material to identify it.

It's only a small thing, but very much reminded me of the holodeck in TNG when I first started using it.

As a VR enthusiast, I'm curious how this might be implemented for on-the-fly integration into a game.

this would be cool for dungeon crawlers period

I think it would make sense in VR as a standalone system with multiplayer which people could use for any type of game or work.

I actually think this sort of thing will become typical within 5-10 years as the systems become faster and we get comfortable VR and MR glasses.

I am currently working on a AAA FPS MP RPG - we're a well funded team with decades of experience in big studios, but not yet announced. The very large world is being generated by a variety of DL & ML algorithms. Thousands of buildings in a city with the interiors of each fully realized. This research work in the paper is almost precisely what we're doing and I suspect the AI team on the game are even now cribbing notes from the paper.

Reply on news.ycombinator.com