I built a low-latency AI companion that plays Skyrim with me | Hacker News Reader

I built a low-latency AI companion that plays Skyrim with me

pantel.is

399 pointspantelisk76 comments

I must say absolutely hilarious video. The "persona" of the dog is great. As someone who is generally pretty "keep your AI out of my art" this looks very fun to play. I can imagine this being cleverly integrated as a primary feature of a game (this must already be in the works). Ideally as a small model able to run locally alongside the game.

The moment where the dog is going on about "something foul in the air" as the player is attacked by a wolf ("F--- dude you could have warned me!") was great comedy.

I get it is fun to play for an hour or so then you inexorably encounter behaviors that are too complex to handle, e.g. you try to sneak in on an enemy but get spotted because the dog started talking or barking randomly, then suddenly wonder how much "it" actually gets it. You then start with less and less complex plans... until you realize that direct commands or shortcuts are more reliable.

I do think it is an interesting exploration but the jagged frontier makes it really challenging to know what will consistently work and what will not, to the point that I bet (maybe pessimistically) one will be gradually less daring with creative plan with their "companion" simply because they can't trust it.

I feel like that's the benefit to the creator having made this a dog (even if a demon-possessed dog). As intelligent as a dog can be, can we ever truly understand why they do the things they do?

The Uncanny Valley effect would be too big if a human companion randomly started talking to you while you were sneaking up on an enemy. But a dog? You'd be pissed off at it for a bit, and then forgive it because it's a dog.

"But a dog? You'd be pissed off at it for a bit, and then forgive it because it's a dog."

Some grumpy old school hunters (in less civilized areas) would literally shoot such a dog though for barking in the wrong moment.

Except it is not a dog. It is an AI. It’s a character in the game.

People often love to fus ro dah Skyrim companions off of cliffs because they can be incompetent annoying little dumb things that won’t shut up.

The character taking on canine form won’t suddenly make gamers more likely to put up with it messing about their gameplay. Thats just not how things work.

> The character taking on canine form won’t suddenly make gamers more likely to put up with it messing about their gameplay. Thats just not how things work.

source?

Kinda in the same vein, been reading/listening to Dungeon Crawler Carl lately. One of the main characters is a tortoise shell cat that gains sentience. One of the more fun aspects in the early books is the main character Carl dealing with that cat being loud, not aware of her surroundings, and making it difficult to sneak around an RPG style dungeon.

Also the author really gets the "tortitude" right.

Princess Donut has insane CHA, sneaking is not the move. Just have her run in and charm everybody.

Yeah makes sense this is what I said about AI and coding. No way it could do anything complex.

Now AI does all the coding for me, I couldn’t have been more wrong. You will be wrong too.

Yeah, so the dog is derpy today, but the possibilities in the future are going to be awesome.

Indeed. Can’t wait until they sell me a subscription to a game that I will just look at while an AI plays and has fun for me. Whee! Awesome times ahead indeed.

Or maybe it won’t play the game for me but will do whatever it wants, mess up about my gameplay because it inferred it should do something I did not order it to do.

Either way, awesome! Lovely gaming experience. I remember being a kid and thinking “wouldn’t it be so much more fun it if I didn’t have to actually play the game?”.

you lack imagination. AI will not play the game for you.

It will construct the game around you as you play. It will customize the game according to you and the situation dynamically.

The first step is AI npcs and AI generated quest lines.

The next step is the entire world, the entire story, the entire game will be dynamically constructed as soon as you start it up.

This. Far too many people let their pessimism get in the way of possibilities and imagination.

> Can’t wait until they sell me a subscription to a game that I will just look at while an AI plays and has fun for me

"Twitch"

So, this is what I have been thinking (and part of the reason why I want to "cook" it more before shipping). Since I have been building it for so long, I am not sure how much "overfit" it is on my data, the way I speak and the phrases I 'm using.

And of course the opposite too, how much the dog trained me to speak to it a certain way to maximize outcome success.

But then I thought, when people play games they are not using highly sophisticated vocabulary and there is probably lots of repetition since they are always under some form of multi-tasking stress (playing and replying/speaking). So maybe... maybe, the system can adjust itself. Use a big LLM offline to say "user said X, we did Y - was that good?" - then retrain itself.

The decomposer is basically a bunch of old-school embeddings/classifiers stitched together, it can train super fast and doesn't need tons of data. Could the thing calibrate itself to the user? Does it even need to? (because as I said I 'm a datapoint of 1 and I am not ready for the potentially huge stream of bug reports when I ship (add some perfectionism to the mix and you get the idea)).

edit: typos

I've noticed that the main (socially acceptable) use for AI in entertainment seems to be laughing at how bad it is. When it fucks up something obvious that any human would know not to do. When it does something so insane and out of left field that it takes you by surprise and makes you laugh. This seems to fall under that category.

I've noticed this too as a DougDoug fan. He seemed to be genuinely confused by general strong opposition to AI from his audience despite a lot of his content, including some of the most well-received videos, being AI-focused. I think the secret is exactly what you describe - his AI characters tend to be extremely deranged and incompetent which is what makes them hilarious.

>The game runs on Windows, the audio processing and brain runs on my M4 MacBook. It could all run on Windows (provided there is dedicated ~12gb or more gpu ram for it)

I do wonder if this is an avenue for console gaming that might be practical in a few years; AI-centric hardware that might be too beefy or expensive for regular users, but can extend new or existing games. Kinda like the expansion paks of old.

unfortunate that the "ALE" design wasn't opensourced (couldn't find a link in their post) but I would be interested in learning more about the design, in particular what sort of data pipeline was necessary from skyrim to give this sort of action flexibility?

This is the sort of thing that the GPT-Live model would excel at when it launches. I'm not sure you'd need the ALE model at all if they launch Live in the API with acceptable pricing as it solves the latency issues with voice entirely. It's an impressive piece of engineering that hasn't been discussed much yet, I guess because it's only a ChatGPT feature and not accessible to developers. The model can dispatch tools asynchronously while speaking which would be ideal for a game.

What you'd want is maybe some kind of Live model with voice warping so it can be given different Skyrim themed 'Nordic' voices, and then custom tools to interact with the game engine.

This is part of the reason I wanted to announce it even before I feel ready to ship... The frontier models have such crazy velocity (admirable) and moving so fast that I didn't want them to make an announcement and suddenly trivialize all the work I 've put into this, overnight.

The fun technical challenges (that can also act as any sort of weak moat) are being taken away one by one, on an almost weekly cadence now! :)

Indeed, The Bitter Lesson comes at us quicker and quicker now

An 80% good enough solution would be to have it periodically say "Hey, you. You're finally awake."

Wow, what a great demo video. And this is the worst this tech will ever be. Though maybe nobody will ever beat the dog's hilarious persona.

This is a fantastic application of an LLM. What’s great is that the LLMisms fold neatly into “dumb but lovable sidekick”.

So even when it chokes or stumbles on a command, the kind of frustration the user expresses when correcting it feels natural and part of the game even.

The voice of the companion is awesome.

I'm surprised how good the talk is (i.e. the voice recognition). My disabled brother is struggling with good native polish voice to text btw. If you people can recommend anything good or local

I'm really enjoying https://handy.computer/

Runs local and has several models that should support Polish, though I can not personally validate this.

Can confirm this is a really excellent piece of OSS for local STT, I use it in both English and Dutch.

The only thing it has trouble with are brand/product/tooling/person names. You can add custom vocabulary, but then it tends to over-correct and insert those custom words when I never said them. However this isn't unique to local STT, frontier cloud models also struggle with this.

>It creates embeddings from the full text as well as its extracted structure, then semantically combines and compares it with action prototypes. A separate classifier estimates whether the turn is a command, question, chat, clarification or complex request. Everything gets merged together.

Does this mean the prototypes and classifications need to follow what you can actually do in the game? Were they all hand-coded or generated somehow?

This is hilariously delightful, thank you for this during my morning coffee.

Next up: A smart glasses companion that follows you around in the game known as life.

Ive seen something like this in sci fi films.

Hopefully it’s a copper arresting you for walking around pointing cameras at people’s faces without permission.

Video game seem like the perfect fit for llm use and I don't understand why it hasn't happened already.

Why would anyone want to have an LLM generate content in a game when people can barely stomach it generating awful pictures and generic prose?

Who would want to have a non-deterministic, unreliable, out-of-your-control companion in a single player game?

What kind of game designer would find it acceptable to have no control over large swathes of their own game? Who would like to play a game that no one really created?

Games are where art meets engineering to create entertainment and awe. LLMs don’t fit.

Reply on news.ycombinator.com