Comment by Animats | Hacker News Reader

Comment by Animats | Hacker News Reader

Parent Full thread

It is different this time, though. Take a look at this open source project.[1]

This is a system which lets you talk to NPCs in video games. It's a collection of off the shelf components held together by some Python code. The components do this:

- Listen to the user talking and convert speech to text.

- Watch the user's facial expressions via webcam.

- Watch the game, and use face recognition on the game images to determine what character is being addressed.

- Run the user's text through a LLM preloaded with about 30 lines of info about the NPC to generate a reply.

- Generate voice output in a voice generated to match the character's persona.

- Modify the image of the character on screen to animate their facial expressions to match the voice output. This is done on the output image, not by animating the 3D character.

Five years ago, that was science fiction. A year ago, half that stuff wouldn't work right. Now it's someone's hobby project.

[1] https://github.com/AkshitIreddy/Interactive-LLM-Powered-NPCs

I think this is a pretty good illustration of why incumbents are likely to capture much of the value though. Some simple scripts using OSS libraries can do some pretty amazing stuff you'd have previously needed advanced research teams to even attempt.

So the majority of the value gets captured not by companies focused on writing new components that nobody else can match, but by the incumbents with the wealth of proprietary data to feed into the components, or the infrastructure to run the more infrastructure-dependent models at scale, or the customer base to milk for AI-enabled versions of their existing product or selling AI consulting services.

Don't get me wrong, it is cool for indie game developers to be able to procedurally generate NPC conversations. But indie game developers are not likely to capture more value from being able to generate stuff very easily than Microsoft.

But it isn't different. People have been using things like Markov chains to experiment with NPC dialogue for well over a decade.

It just never got widespread adoption because it's just not interesting, and LLMs are no different here. The dialogue is still empty, despite being deeper and more grammatically complex than previous attempts.

If every farmer in an RPG hands out the same "collect 20 bear asses" quest it doesn't matter if they all have "detailed" randomly generated backstories and can opine about the game world, real world philosophy, or the 2024 US elections.

I actually think it makes a world of difference to opine about the game world. It's so much more immersive.

Have you ever gone to a living history museum? (Old Sturbridge Village is one example, my favorite I've been to). All these people in character, able to talk about the period, it makes for an amazing experience.

In traditional video games, if we try to or even accidentally push any deeper, we see the cracks in the universe. "Oh, I spoke to this person again, and they said the same thing to me." AI can help fix those cracks, and fill them in wherever the player ventures.

This certainly doesn't change Fortnite, but I think it could change immersive RPGs and MMOs.

"Living History" is a well crafted written experience, not procedurally generated slop.

The issue here is that LLMs can only act in-character if the world has already been built and written, if the prompts are so pre-chewed that you may as well just write the dialogue directly and get even better results.

Take Solaire of Astora. He's an interesting NPC not because of any depth of the dialogue, but because of how well in-tune he is to the world and game itself. A true believer in the old god, a beacon of optimism in a depressed dying world, and someone who sets the tone of the co-op multiplayer to be silly and fun.

You can't get that out of an LLM.

“‘Living History’ is a well crafted written experience, not procedurally generated slop”

Having known people who lived/worked at a living history museum, their experience was much closer to improvisational comedy than a scripted interaction. Sure, they were riffing on their historical knowledge instead of cracking jokes, but it was not scripted.

> It just never got widespread adoption because it's just not interesting.

True. There have been NPC systems where the NPCs had motivations and a life of their own, even when no one was around. Those haven't helped gameplay much.

The current problem is that LLMs don't know enough about the game world. Recent progress on that.[1]

[1] https://arxiv.org/abs/2304.03442

Reply on news.ycombinator.com