Hacker News Reader

Comment by A_D_E_P_T | Hacker News Reader

Okay, this is probably the right place to say this: You can't play D&D with GPT-3/3.5, but you absolutely can with GPT-4. GPT-4 is a solid DM.

GPT-3 couldn't remember what it said 30 minutes ago, and was basically incapable of planning, so unless you want a simple dungeon rougelike-esque adventure, it'll lose track of things and start hallucinating. It will also sometimes forget the rules.

My initial impressions are that GPT-4 is nearly as competent as a human DM, and much faster and more flexible. Also, unlike GPT-3, it will roll dice for you, and it knows the rules just as well as most players.

These maps + GPT-4 could = infinite campaigns.

Perhaps it's just a matter of time -- and just a few years, at that -- before generative text and graphic AIs merge, and games like Baldur's Gate can quite literally be made on the fly.

View on HN

Played a solo-campgain with GPT4 for 1-2 hours yesterday, summary:

- Created character sheet for me, at desired level with spells and abilities

- Was able to add a cat to the game upon request

- Took hints.

- Was an outstanding writer, far better than any DM in terms of sensory descriptions and articulation

- Had to ask it to only tell 5 paragraphs at a time

- Performed combat rolls and math, all apparently correctly for a long while

- The first 20 rolls were all greater than 10, asked it give more random rolls and it seemingly did for a while

- Added elements on the fly consistently without losing track mostly

- Used puzzles and challenges and created a sense of danger. Adapted well on the fly to unusual character behavior (guessing an unreasonable answer for a puzzle solution).

- After a few hours it started confusing classes in my party (I think there is some context window). It forgot I had agonizing blast after a few hours.

- Sometimes fudged distances

Overall for a DM with infinite patience and an entirely customizable campaign on the fly with no need to plan/recover it seems promising at first. But some sort of prompt engineering will eventually prove necessary i think.

It might be possible to 'extend' the context window by summarising important information from time to time, to refresh it's memory.

Did it feel fun?

Definitely did for me, but I also wonder if I learned the limitations/weaknesses of the AI too much that it would break the immersion.

It's dizzying to think about how immersive these things will become after the next however-many iterations that are foreseeable.

When they understand and can manipulate psychological "archetypes" (or whatever they find in the structure of our minds) and we let them make our kids' cartoons and games...

Also, “asked it give more random rolls and it seemingly did for a while” seems a problem area to me, not as much for that “for a while” as for the fact that it complied. What happens when you ask it to give you more favorable rolls or to let you kill a monster with your bare hands?

This is a case where Asimov’s laws of robotics would have to be bent a bit. If you ask an AI to let you win a single-player game, should it comply, or argue that letting you win will harm you?

Human DMs will frequently respond to a player's mood, and will fudge rolls to be more (or less) favorable to the player if they feel that players aren't having fun. The only real difference here is that GPT couldn't respond to the mood, and had to be told to fudge the rolls.

Had to be told NOT to fudge the rolls, right? If it seemed like the rolls were skewed in the first place?

Nobody writes about a perfectly average dnd session, so its training data probably has less than usual “average” rolls, which might result in it not understanding that dnd can be average sometimes.

>This is a case where Asimov’s laws of robotics would have to be bent a bit. If you ask an AI to let you win a single-player game, should it comply, or argue that letting you win will harm you?

Game AI (single player in particular) is all about providing the player with openings and ways to win and feel powerful without them realising that's what is happening... all the way from NPCs coordinating with one another (and barks to make it clear what they are planning so they can be ambushed or stopped), to "Director AIs" that work on rules of tension and release to make sure the player's having the most fun and not being overloaded.

> Asimov’s laws of robotics

This comes up every time AI is mentioned and I feel like people are drawing entirely the wrong conclusion from Asimov. First and foremost he designed his laws so conflict was inevitable. The laws are designed to produce interesting fiction, not to be a template to follow in real life. They are as much a manual as "1984" is, that is - not at all. Asimov's laws should have no bearing on AI in reality except to inform people to do something different.

So broadly I am agreeing with you - AI should follow laws that are more rich than Asimov's made up universe. The part that irks me is that people even consider using them at all. It's fiction, not reality. The consequences of an action in a person's imagination rarely match the consequences in reality. The entirety of science is an effort to produce word-based imaginary systems and processes, whose outcomes match the outcomes in reality. You cannot just substitute one man's imagination for the missing science of AI reasoning.

1) is there a sample transcript? YOu say OUTSTANDING WRITER and I think GRR Martin.

2) an interesting runaround copyright. GPT4 probably has every "d20" product and others ingested for processing and use.

Look people, here's the ultimate test: write Winds of Winter and Dream of Spring for us.

Between all the reddit /r/asoiaf and fantheories plus all the other works of fantasy, it should be able to do some facsimile.

Honestly GRR should be cranking out a chapter or two a day with GPT4 help.

There should be entire eras filled in by GRR with edited GPT4 output.

> Look people, here's the ultimate test: write Winds of Winter and Dream of Spring for us.

GPT-6 or 7? That would be great.

At some point, each of us can have our own ending to the series, with as many additional books as we want.

Then all recreated as a photorealistic VR show. Viewed from the perspective of the respective point-of-view character of each chapter.

Then a complete role playing version where the entire plot is contingent on our own choices, ... all the Martin and HBO's Westeros world building paying off. Every time you die, you are dropped into another interesting character in the vicinity of one of the latest fulcrums of the Westerosy future.

Massively multi-player, in real time.

This would have seemed speculative fantasy (hah!) not so long ago. But now perhaps, inevitable in the not-too distant future.

The dystopia of having no work, but being endlessly entertained, has never looked so good.

> YOu say OUTSTANDING WRITER and I think GRR Martin.

Wow. G.R.R. Martin, much like Stephan King, writes to the least common denominator, which is about a 5th or 6th grade reading level. I challenge you to find a complex sentence in either of their catalogs, which is not to say their ideas are bad, because they're not. But their writing is absolute shit. Shakespeare, Dickens, Hemingway, Tolstoy, Joyce, and Camus are all outstanding writers, while the only thing that made Martin's writing bearable is H.B.O.

I wonder if there's something that you can do as you're nearing the end of the context window to summarize the "state" of the world so far and put it into context so that the model always has context on the most important details. I'd imagine that you'd lose some of the minor stuff by compressing context this way, but, hey, humans forget minor details too.

The way I would build a DM with GPT4 is with a few external tools: the character sheets, random dice, and perhaps other things should be available on the side, and fed to the chatbot when needed. Same for a summary of the campaign, so that previous events are not forgotten.

When you think of the content of the books in Baldur's Gate and the sheer amount of human hours poured into something just to give background to the game, this is going to save indeed a lot of time.

But on the other hand, just like open world was a terrible curse with hundred of games feeling empty, I'm afraid we will see a wave of games with a lot of generated but eventually uninteresting content because this is so easy to do.

After all, good content is coherent, and filtered. It's not just about producing a lot, it's what you don't put in the game, and what you do to glue things together.

I sure GPT will eventually be better at it than humans, but for now, having a human veto is necessary to have a good noise/signal ratio and because GPT is "good enough", being lazy is tempting.

But yeah, I can confirm GPT4 is already quite good at it and will only improve from this point.

This is going to be very interesting years.

Edit: I just made it write a full video game scenario, using BG as a competitor for basis. At first it was average, but with directions, it refined it and became quite good. Some kind of steam punk guild called the Navigators reading the wind, living in floating islands and a girl rediscovering the earth bellow the clouds and rebuilding old aliances with lost civilizations, plus a politician leader trying to prevent that and even a mentor to help her. Not bad.

I would play this, and even read it.

You can see the inspiration from "The Pillars of the Earth", "Gunnm", "Baldur's Gate" and "nausica" proving again it's mostly assembling known stuff.

But a cake is mostly assembling known stuff and it's still good. The "last of us" is just a good take on something that has been done a thousand times.

So why not?

What's missing is overt structure. More or less the same reason as to why AI can't draw hands - there is no "understanding" of the fact that a hand is a rectangle with five sticks coming out of it in a specific arrangement. For a human, that is all that's needed to recognise a hand, i.e.

            |   |
        |   |   |
    |   |   |   |
    |   |   |   |
    |   |   |   |
    |   |   |   |
    +---+---+---+
    |           |
    |           |
    |           |    |
    |           |    .
    |           |   /
    |           |  /
    +-----------+-+

This is a (very crude) hand. This registers as a hand for you, because it has the structure of a hand. This is less similar to a photorealistic hand than a 6-fingered AI hand, objectively, however it fits the human understanding of what a hand is supposed to be, because it has the right kind of shapes in the right relations to one another. In the same vein, if we could get the AI to understand the structure of a story, it should be able to DM session by session, starting from a high level abstract idea and refining it based on player actions, incorporating player input into the game. I'm not sure if hierarchical abstract reasoning should be forcibly added to the model, or if it can be emergent in a specific configuration, but if we figure out how to build an AI with that, then it should be able to produce something a lot more sensible.

I had a fun conversation with GPT-4 about your ASCII hand. You're absolutely right, it had a very difficult time and thought it was a city skyline, game or puzzle diagram, and a scale, but after a few hints, it got it:

"Considering the hint that the image is part of a human being and the diagonal line is similar to the vertical lines, it appears that the image might represent a hand, specifically an open hand with fingers splayed out. The vertical lines represent the fingers, and the diagonal line represents the thumb, which is extending out at an angle from the square (representing the palm of the hand). The image is a 2D, stylized representation of a human hand.

With the provided hints and context, the interpretation of the ASCII art as a hand seems more plausible. If you have more hints or information, please share them, and I'll be happy to refine the interpretation further if necessary."

This technology appears to show great promise, but is clearly still in its infancy.

            |   |
        |   |   |
    |   |   |   |
    |   |   |   |
    |   |   |   |
    |   |   |   |
    +---+---+---+
    |           |
    |           |
    |           |    |
    |           |    .
    |           |   /
    |           |  /
    +-----------+-+

Oh, hey, a nice steaming cup of tea!

...

Uh oh.

>> For a human, that is all that's needed to recognise a hand, i.e.

Blast it! Cover blown.

:poof:

> Also, unlike GPT-3, it will roll dice for you

I'd question the fairness of those rolls.

But yeah, DMing seems like an excellent fit for these models. But you probably need to track hitpoints and stats separately, GPT-4 can't do math either. Or I guess it not doing math might be a feature, that way it will tell you what reasonably happens with fuzzy logic, might be a more immersive experience.

It would be interesting to see the dsitribution of dice rolls when asking GPT-3 and 4 to roll a dice repeatedly.

I hope they didn't train it on transcripts of XKCD.

https://xkcd.com/221/

You don't need it to be fair, you need it to be fun. Most video games play with randomness so that the distribution serves the gameplay, sometimes bypassing completely the RNG to balance frustration/satisfaction.

GPT can learn do to that: not providing true randomness, but the illusion of it to serve the game.

Players will detect drama being injected into their randomness (unless it is very gently and rarely done), and over time they will start to be able to predict dice rolls [edit: or even learn to shape their responses to trick the model into generating the rolls they want.] That will ruin the game.

Xcom 2 make hits more likely the more you miss because fair proba was frustrating the players when they missed a shot with 95% of success.

So I'm not sure you are right.

A 95% shot will never miss in Xcom 2 unless you play on harder difficulties. It multiplies hit chance by 1.1 so 91% is 100%.

So that's not removing randomness, it's just grade inflation. I suppose it's also secret range narrowing, since an 3% improvement from 92% to 95% is actually no improvement at all at 1.1 * (random) capped at 100%.

> My initial impressions are that GPT-4 is nearly as competent as a human DM

I wonder what kind of campaigns we're talking about here. I mean, if it's just mindless modern encounter-based design, sure, I can believe that. Kicking down doors, rolling for initiative, GOTO 10.

But even mildly more moderate DM work like re-jiggering set up monsters to react to the actions of the players (in e.g. a Caves of Chaos type of setup)?

See, that's the thing.

No matter how much effort you put in to prompting and coaxing it, GPT-3 really couldn't manage anything but the simplest encounter-based game design.

There were things that GPT-3 did really well -- it could write you a 600-word essay on the characteristics and atmosphere of each chamber or passage you find yourself in -- but it couldn't stick to a narrative or engage in anything resembling long-term planning. So, ultimately, you're left with something quite strange and dreamlike: A campaign that's vividly -- even hyper-realistically -- described, but where nothing makes sense and there's no logical continuity over time.

GPT-4 knows what it's doing, though. It's very different. As far as I can tell right now, it combines the outstanding descriptive capabilities of GPT-3 with the ability to (a) remember things it has previously said and establish logical relations over time, and (b) make sure that the campaign appropriately mirrors a "normal" D&D campaign. GPT-3 would frequently mess up and have you fight dragons at Level 1; GPT-4 more appropriately sticks to goblins and bandits. And, when it comes to tactics, pacing, etc., it's really quite good. I suppose it's a little bit conventionally-minded, but that's to be expected.

Almost scary to think that it'll only get better from here on out, because GPT-4 is damn near "good human DM" tier, and already much better than a bad DM.

I've been extremely impressed with GPT-4 so far, generating a campaign for my kids. The encounters it created were very well fleshed out (monsters, minions, stats and abilities for all, including very flavourful lair actions for bosses). It's given me NPCs and sample dialogue for each, locations, and a central storyline that is well-threaded throughout. I was genuinely astonished even at its first attempt. I don't know if it could stand up on its own as an interactive dungeon master, but as a tool I am loving it.

What does mindless modern encounter mean? My sense was that modern players are more plot and narrative focus whereas old school players were down to stuff like roll for a random encounter while traveling and actually play out a fight with… Wolves! That didn’t matter at all to the plot. It was just what you did.

Edit: but I could be wildly off here?

My feeling is that all this varies far more between groups and possibly even geographic regions that anything else. My memory of the late 90s/early 2000s was that storytelling was the focus, dice where used sparingly and that the "rule of cool" was the only rule that mattered. All much more so than what I see today.

I've however also spoken to other people, living in different places, and they remember the same time period being all about rules lawyering and "war gaming".

I've also observed a Europe/US divide where the US tends more heavily towards "war gaming" focused role playing with physical maps and moving miniatures around on a table, while Europe tends towards more story focused role playing. But I could be imagining that.

I have been seeing the opposite myself.

In the 90s and 00s it was almost always about the minis and tactical gameplay. DnD 3.5 was a lot more numbers heavy so it kept a lot of the story people out of the game.

Now DnD 5 the numbers have such a gentle, predictable curve with bounded accuracy and damage and stuff, you may as well ignore them because there's nothing exciting about rolling the dice anymore.

So the story is a lot heavier focus, the character journey. I never used to encounter people who got upset if their character died or groups who had the mentality that characters must always fail forward for the sake of the story.

That stuff is everywhere now.

Another aspect I just thought about was that in 90s and 00s there was a lot more diversity of games that were popular. Most people I knew played at least 2 or 3 different games with very different styles and I knew lots of people who never played DnD. Now everybody I talk to seems to play DnD only.

Just the fact that the tile is "D&D map makers..." instead of "RPG map makers..." is very telling.

Also true. Everyone I knew would have a game of Shadowrun or World of Darkness or whatever else if you wanted. At the very least there were Pathfinder groups.

But now I don't even see Pathfinder out there much.

I think DnD podcasts and webseries and such were incredibly successful in cementing DnD as the only game in town. They brought in a ton of new players who have only heard of DnD and only want to play DnD.

My group has gone through a few systems and we still refer to it as a "D&D" group and just understand that as referring to TTRPGs generally. I wonder how many there are like us.

Note that "DM" was mentioned, so I'm strictly talking about D&D here, other games and genres focus a lot more on the narrative aspects of RPGs in a planned manner (NB: Although I personally would suspect that the "AI"s we got might do even better at PbtA games than D&D)

Here, at least since mid to late 3rd edition, we got a very strong focus on combat "encounters", where you got pre-packaged situations where combat would arise. A combination of monsters, location and possibly events happening during that time. Monsters might even be built for this specific situation, even though their names are generic enough (i.e. a "Orc Battlescout").

Your dungeon FSM gets to a certain state and this encounter gets triggered.

Whereas the canonic ideal of older games was a more fluid setup. Often described as "Combat as War" vs. today's "Combat as Sport". Sure, played very simply, you got an even less sophisticated environment: Room #13 has 4 orcs, guarding a chest, which has 380 copper coins. Kick down door, fight orcs, spend coins on torches for next dungeon. But no experienced DM sees a dungeon that way. Because next to that, you got room #14, where the orc chieftain is preparing the prized pigs they stole from the goblins in room #27, which are currently sneaking up to get it back. Then, as you say, you might have your random encounter, so while the players are heading for the orc room, wolves attack (the goblins aren't guarding the wolf-forest-gates, the DM muses). The fight makes a lot of noise. The orcs look. The orc chieftain looks. The goblins use that distraction. The characters might notice that or not, etc.

Even rather simple setups can achieve rather complicated results, if you're not emulating a 12 year old mastering their first adventure where they just picked monsters from the tables in their Dungeon Masters Guide that are appropriate for a level 2 party of characters.

That's a bare minimum where I'd consider a "AI" DM-like, otherwise you mostly got a NLP interface on gameplay that I could've had with the old SSI Gold Box games on an C64.

I spent some weeks trying to make a GPT-3 D&D GM, but indeed it just won't work. I haven't tried GPT-4 yet but this post makes me want to test it for sure.

The problem I had with GPT-3, most of all, was that it tries to "finish the story". E.g. if the players are on a grand campaign to defeat some big bad boss, then this might happen:

GM (GPT): You enter the tavern, and you see a halfling bartender and a few gnome patrons.

Player: I ask the bartender for information about <BIG BAD EVIL GUY>

GM (GPT): The bartender tells you <BIG BAD EVIL GUY> lives in a castle. You and your party storm the castle and manage to defeat <BIG BAD EVIL GUY>.

I managed to get a bit better than that story-wise, but I really felt this problem with battles, it just refused to do an actual battle. When a battle scene started, it always rushed to "after a long and tiring fight, you eventually come out victorious" or something to that effect.

I run a small D&D group with a couple of friends. I don't know that I'd want to play in a campaign helmed by an AI dungeon master, but the idea of having some help generating plot, assets, etc. when I don't have a lot of time before a game session sounds amazing.

I tend to be an AI pessimist, but this seems like a good example of a way that AI could be used to supplement human creativity instead of just replacing it entirely.

This whole thread has me re-evaluating my prior assessments of "Ender's Game" and the Mind Game it described.

When the book came out I was a child programmer and I had total contempt for the idea that a player could just decide to burrow into the eye of the giant and that the engine would generate a result.

I was used to Sierra games where you'd click an inventory item on something you saw and the game would tell you "there's nothing useful you can do with that."

I actually currently do this ( for cyberpunk red not D&D). What I will say is it has let me be way more open in letting the game go where the players want it. If they decide chasing after some thread that I wasn't even thinking about is what they want to do, I can stall for a bit and let chatgpt come up with some characters, locations, and interesting skill checks.

That's exactly what everyone said about GPT3, two or three years ago

> Before, you couldn't have a chatbot run a D&D campaign. But GPT3 is a solid DM

and then when all the hype died down to acceptable levels, turns out it was not. So forgive me if I stay skeptical until I see convincing evidence.

I am working on something like this, text-based with images. But I don't generate on the fly, I will have big lists (many thousands) of curated text outputs from GPT that get selected with randomization and nested if statements. For those interested, the game (work in progress) is Advinsula: https://advinsula.com/

This has to be the most inspiring comment on HN ive ever seen. I must live to see this.

It is also fantastic to create on the fly scores for Blades in the Dark, surprisingly detailed and able to flesh them out.

Runs for Shadowrun are also possible, really nice. Great for systems where you don't have a lot of published adventures.

> Perhaps it's just a matter of time -- and just a few years, at that -- before generative text and graphic AIs merge, and games like Baldur's Gate can quite literally be made on the fly.

All we need after that is an AI generated musical theme / background ambience / sound effects. This came to mind while listening to https://open.spotify.com/episode/1xs8fxI5ZdaIA1i386IHH2 yesterday

How does GPT-4 remember things this long? The memory of it is just 8k / 32k tokens long, right? Seems you would run out of tokens quickly.

Yes! with enough generative AI capabilities (Star Trek) Holodeck can be achieved.

Yes but will it be able to make the holograms come to life and take over the bridge?

But can GPT-4 suggest tunes for Nero to play while Rome burns?

Yes

People dream every night for an hour or two. If you can remember your dream and type it into ChatGPT can it interpret it? "Yes" is not an answer.

Yes, it can. Why is Yes not an answer?

Can confirm, saw somebody post this here the other day: https://www.youtube.com/watch?v=H-89vnqxkFg

I don't think it is a matter of years, it is a matter of months.

in my eyes the big problem is that these model are purely statistics and are unable to apply logic

Reply on news.ycombinator.com