Mistral-7B Playing Doom

github.com

48 points·tosh··10 comments

I've done some experiments with a little camera-equipped robot navigating by prompting a LMM with stills and producing captions that include military-style o'clock directions. Some promising results with a weekend of tinkering. I'm sure one could play Doom that way, too.

I'd love to give my robot vac water gun abilities so it can defend itself from cat attacks. One day.

I always thought that in terms of getting the current generation of LLMs to do useful actions on a computer, we need to go back to console applications since they can't yet interpret GUIs but are perfectly fine with text. This seems to be an example.

I'd be interested to see how an LLM might go with an ancient console based spreadsheet or word processing app. Could they then replace humans using modern versions of those same apps?

Related / previous discussion

https://news.ycombinator.com/item?id=39813174

Would be really neat to extend this to have Doom on an LLM. Much of AI has been enabled by gaming, and this could be the crowning synergy.

LLMs aren’t computers. What do you mean doom on an llm?

"You are a cacodemon. Your objective is to kill the Doom Slayer using a combination of your ranged ball lightning attack and a bite at close range. For each frame of input, select the response that represents the best course of action to achieve this goal at the time of that frame."

Oh, so just a doom themed text based adventure?

Didn’t AI dungeon already do that?

More like plugging an LLM in in place of Doom's enemy AI.

Oh, that's what the Doom-playing program sees as a representation. That's very clever.

Reply on news.ycombinator.com