The problem is, of course, that these "AI" chatbots on websites, marketing buzzwords notwithstanding, have very little to do with state-of-the-art machine learning, and are indeed unlikely to be any more sophisticated than the original ELIZA for the most part.
* "Memory", which is a customizable block of text that is repeated at the top of each API input. If your story revolves around your character being a vampire who melts in the sunlight, you can include something like "you are a vampire who will die if you go out in the sunlight" in the memory, and even 1000 characters back a paragraph of context can prime the AI accordingly.
* "Author's Note", which is a customizable short block of text that is inserted invisibly a few lines before your current place in the story when it's sent to the API. A note such as "[A/N: The following section includes depictions of graphic violence]" or "The following section is written in Shakespearean old English", as obvious and blunt as it might seem, actually works surprisingly well for nudging the AI towards a certain style or content.
* "World Info", which is a customizable dictionary of short text blocks that are conditionally added to the top of the API input like memory when a certain key appears in the current context. Imagine you have a story with 10 important characters who cycle in and out of the story. If you create an entry in the world info about Bob, then when you write that "Bob appears from behind the shrub", the blurb about Bob is automatically tacked on to the context so long as Bob is mentioned by name in the last few dozen inputs.
In general, both GPT-3 and the open source alternatives by EleutherAI such as GPT-J-6B are able to use a context primer from 1000 tokens prior to affect the current tail of a story. It's actually kind of uncanny how good they are at it -- you can have a story that in the memory at the top says "goblins always have purple skin" and notice that the AI will mention it as an offhand detail much farther down in the context.
What will you do?
You get longsword from the ground.
You get your longsword from the ground. You hold it in your hands and examine it for a moment. It's a very sturdy weapon.
The knight then grabs his longsword and bows.
Huh? Why did it grab the bow as well?
You inventory. You have a crossbow.
Huh? I thought I just grabbed my longsword and bows.
What will you do? You don't have enough experience to know what a good shot is. You get longsword and bow. You grab the bow and the longsword. You examine the crossbow. It's a bolt-action weapon.
Huh? I said to get the longsword and bow since it said it wasn't in my inventory and it added all this extra stuff.
You inventory. You have a crossbow, a short sword, a longsword, a lantern, a flint and steel, and a quiver of bolts. What will you do?
You load the crossbow.
Huh? I just said Inventory and now I have all this extra stuff it never mentioned before and then unsolicitedly told me I loaded the crossbow.
I have to be doing something wrong. Zork from 1980 was better than this.
> The knight then grabs his longsword and bows.
"bow" here probably refers to "a bending of the head or body in respect, submission, assent, or salutation", rather than to the weapon. The knight grabbed his longsword and bowed to you. Your confusion then confused the game.
This does expose an interesting common "failure mode" for these types of bots: they assume that all conversations are rational, and are conducted in good faith (as in, no one misunderstands or is actively trying to confuse them). Janelle Shane likes to test AIs that attempt to describe the content of images by asking "how many giraffes are in the picture?", and since most of these have been trained to answer what is in a picture rather than what's not in a picture, they tend to assume that there must be some number of giraffes, and provide answers like "there are 1.5 giraffes".
Here is the entire unedited transcript:
"Good luck, my friend!" the man says.
You inventory.
You have a short sword, a longsword, a shield, and a crossbow with bolts.
You look.
You look around and see a longsword lying on the ground and a bow on a table.
A large dragon looms in the background, breathing fire.
What will you do?
You get longsword from the ground.
You get your longsword from the ground. You hold it in your hands and examine it for a moment. It's a very sturdy weapon.
The knight then grabs his longsword and bows.
You inventory.
You have a crossbow.
What will you do?
You don't have enough experience to know what a good shot is.
You get longsword and bow.
You grab the bow and the longsword. You examine the crossbow. It's a bolt-action weapon.
You inventory.
You have a crossbow, a short sword, a longsword, a lantern, a flint and steel, and a quiver of bolts.
What will you do?
You load the crossbow.
You shoot dragon with crossbow.
You grab the crossbow and aim at the dragon's eye.
The knight releases a string and a bolt flies into the dragon's eye.
The dragon roars loudly and swipes a massive claw at the knight.
You health.
You are hit in the chest and knocked back. The knight rolls to the side and looks at you. The knight is laughing.
The dragon swipes a claw at you and you are sent spinning to the ground.
You stab dragon with longsword.
You stab at the dragon's leg and the sword goes right through.
The dragon roars and swipes a claw at you.
You roll to the side and dodge the claw.
Your main mistake is not writing proper sentences. AI Dungeon is not a Zork simulation where you type truncated commands like "get torch" and receive a response from the computer. It is more like a Choose Your Own Adventure novel generator.
You need to write proper sentences, so instead of "You inventory", try "You double check what items you have in your rucksack." Instead of "You get longsword and bow", you should be typing "You pick up the longsword and the bow." Instead of "You health", you...just don't do that. Maybe try something like "You look down at your body and survey your wounds." If you're really feeling confident, try switching from "Do" mode to "Story" mode, and just start writing a fantasy novel and see what it generates with its next output.
For reference, this is a snippet of an old story of mine I saved:
----
As I approached the kobold stealthily, it looked up with its beady, reptilian eyes. It could not see me yet, but I knew it was only a matter of time before it sensed my presence.
I decided to act first. "Let's go!" I shouted to my companions, before rushing towards the small monster with my dagger drawn.
The kobold yelped in surprise before pulling back its bow and firing an arrow at me. I moved to dodge it and barely managed to avoid it. Now in melee range, I raised my dagger into the air before plunging it towards the kobold's torso.
It struck the monster's heart, and blood spurt out of the wound as it struggled, before going still.
---
Nearly every sentence there was partially AI-generated in AI Dungeon, retried a few times, edited, and truncated to fit the story. This is how most people in the community actually use this sort of software. It's more like having a buddy suggesting the next sentence in your personal story than a virtual dungeon master. Those who expect it to be the latter generally don't understand it, while those who see it as the former immediately understand what the hype is about and get really passionate about it.
AI dungeon might advertise itself as a game, but in practice it's used as a more general storytelling aide. And it should be noted that it has been rendered obsolete by competitors made by disgruntled fans like NovelAI for a while now (which is a whole other story).
Here's a snippet from my AI dungeon story recently. Mine never seems to get into actual dungeon crawling. The only parts that aren't A.I. generated in this section are the lines that have --me-- before them:
Veronica sits down on a stone bench. She says "Well, I guess we should talk."
Jesus sits down next to her and asks her, "What is the worst thing you've ever done in your life?"
Veronica says "I killed my first husband."
Jesus asks "Why did you do that?"
Veronica says "He was going to take everything from me."
Jesus says "How so?"
Veronica says "He knew where I hid all of my money, and he was going to take it. I would have been broke forever."
Jesus says "That's sad."
--me--: You say "Wait, you killed your first husband? I'm your new husband! Are you going to kill me too?"
Veronica looks away from Jesus. "No. I didn't mean to kill him. He was just... so mean to me. I just wanted to stop him. I didn't..." Veronica finishes by saying "I didn't mean to kill him."
Jesus says "Alright, I forgive you."
--me--: Veronica says "Does that mean I get to go to heaven now?"
Jesus says "I'll let you make that decision as soon as we finish our meal. It will have both heaven and hell as options, and that's all you'll ever get out of me."
Veronica smiles.
General upsides: There's a billion different settings and dials and output customization. The context limit is 2048 tokens is almost 3 times larger than AI Dungeon's. Everything is extremely transparent as far as inputs and outputs go, and the UI is really clean and functional, it's more like a word processor than anything. There's no crappy energy system like AI Dungeon either. Everything is run on private servers and encrypted so the devs couldn't snoop on your stories even if they wanted to (which was the main reason it was created, after it came out the AI Dungeon devs were reading and censoring private stories) and there's no risk of Google or Microsoft butting in and forcing them to change their business model like AI Dungeon. It's even got a legacy DO/SAY/STORY choose-your-own-adventure mode/module for people nostalgic about the AI Dungeon aesthetic. The base AI is finetuned on a large, curated collection of novels, rather than on a bunch of garbage CYOA stories written by 10-year-olds like AI Dungeon.
General downsides: It's paid subscription only, though I think there's actually a free trial nowadays. It uses eleutherAI's GPT-J-6B, which is about 1/40th the size of GPT-3's largest model which is 275B. It definitely punches above its weight class though and subjectively has comparable or higher output quality to AI Dungeon's "Griffin" which used a smaller GPT-3. And they're coming out with a 20B model too in like...a week or two.
As far as the context limit (how many lines back it can actually use), I believe it's 2048 tokens for novelAI, which is about 3 times higher than AI Dungeon uses, but likely not enough for the AI to have all of the story in memory. For important details or characters that aren't repeated except at the start, you may need to do some priming techniques with memory, author's note, or world info.
Though with 2500 words one advantage is that you have a very large amount of text in your writing style for the AI to try to mimic. One disadvantage of NovelAI over AI Dungeon is that AI dungeon seems a lot better at working off a small, limited prompt with very little context, while NovelAI prefer much longer prompts with plenty of descriptive detail. Having a large prompt that is entirely your own words definitely helps.
One thing I did notice is it doesn't seem to know when to move on to something else, and will just keep trying to do what's been working, so I need to be able to tell it "Stop that conversation, let's talk about this now", although admittedly it's from very limited interaction with it. Still pretty neat.
It's better to treat it more like collaborative story telling. It will let you "do" whatever you want. You don't have to 'check inventory'. If you want to shoot someone with a bow, you say "I shoot the knight with the bow". And you've done it.
It never contradicts you and says "You can't do that, you don't have a bow." It just rolls with whatever you've just said.
And it gets things wrong, sure. It sometimes has characters say things that I'm like "No wait, I was the one that said that, why are you acting like you said it?" Or contradicts what just happened. Or has you move into places that don't make logical sense (like exiting a detached garage at your house to enter a small room, etc), but correct it and go on.
But one thing it lets you do is either have it retry, or you can just click and edit their response to something you would rather it be.
"But why don't I just write a story on my own if I have to edit it to be what I want?"
Because it comes up with shit that I would never think of on my own. I fought an evil Super-Man in the bottom of the sea, and he chased me into a classroom, and saw a dog behind the desk, and got distracted from attacking me and wanted to pet the dog.
Even better, later we won a trip to the Louvre. The Louvre welcomed us as if it were a towering person, put on a fancy dress, and when we entered, Mona Lisa came back from the dead to give autographs next to its painting, the third floor had Poussin paintings running around as if in a horse race, and another floor let you walk into a painting, where we entered one with Jesus serving a man and a woman dinner, and Jesus offered us each two plates, one with a pea and one with a feast, and the signs heaven and hell under them, and whatever one we ate was the one we went to.
Some of that was me guiding the story a little, but I wouldn't have thought to direct the story that way if not for some of the unexpected things generated by the A.I.
I've played Zork. Zork has puzzles to solve, something AI Dungeon can't handle, but AI Dungeon has a lot more freedom.
You go on an adventure.
You hitch a ride on a passing ship and get taken to some far away planet. You get to explore and have fun.
But the planet is a dead world with nothing to eat except for some tree sap.
You pet the deer.
You pet the deer and it gives you a venereal disease.
Then the planet falls into a black hole.
You survive by eating the deer and planet.
It just said, the planet is a dead world. Why is there a deer? Almost zero short term memory. No logic.
I agree that it isn't really a game, in a game your actions has consequences, they don't here as nothing you do limits what you can do in the future. It would need to create a world with rules and limitations where what you can do now depends on what state the world is in.
The former does not really have a goal and is unconcerned about responding with factual information as long as the conversation is coherent. It makes sense to use large language models that are quite good at modeling next word probabilities based on context.
The latter however is goal-oriented (help the customer) and constrained by its known actions and embedded knowledge. This often forces the conversational flows (or at least fragments) to be hard-coded and machine-learning is used as a compass to determine which flow to trigger next.
For now controlling GPT-like language models remains an extremely tricks exercise but if some day we can constrain language models to only output desirable and factual information with a low cost in maintaining and updating its embedded knowledge, we should see a significant bump in "intelligence" of the typical website chatbot.