Comment by AeP6cheo | Hacker News Reader

Comment by AeP6cheo | Hacker News Reader

I really hope this tech will be miniaturized at some point.

I hate to be that guy, but when I saw the article the other day that ended with "plot twist, this article was autogenerated with GPT-3!", I was not impressed, mainly because it looked like content farm's content to me, conveying no information. It basically looked like an incredibly costly spam tool.

But then I thought of possible applications. Give humans a new tool, and they will amaze you, I'm sure we'll see cool applications of this tech in the future (and probably horrible ones too). One thing that I could think of where it would totally rock : games. If games were able to use such tech, their game world could be filled with casual discussions rather than the developers needing to fill everything. This means that NPCs in a game could have ever changing discussions, and even answer to the player about the subject they're discussing. Their discussion could even change based on what is happening in the world or what the players are doing, even to things with the smallest impact, or what is happening directly around them at the moment, without devs needing to script it all. That would be awesome.

But yeah, this won't happen easily unless the model can be embedded and shipped on local computers.

Hm. GPT-3 is trained on internet data from the real world. Your NPCs are in the gameworld. I guess you wouldn't want NPCs to ever reference anything 'real' for fear of breaking immersion (not to mention political backlash if the model grabs the wrong thing!). However, if you limit your corpus to the gameworld that's nowhere near comparable to the real dataset, and someone would have to retrain it.

There needs to be a way of absolutely limiting it, from what I've seen the prompt can't do that with 100% success. A blacklist would be useful in some applications (people tolerate their search engine screwing up once in a while), but not in games. Still, maybe we can accept limited performance by limiting to the gameworld + some manual texts. It's not NPC dialog is literature-level anyway...

The main thing about GPT-3 is that they wanted to demonstrate one-shot fine-tuning and succeeded at it.

So the model can be transformed to output part-of-speech words, dependency grammar trees or named entities in input even if training data is sparse. Similarily, you could fine tune it to produce game lore and then see how it works for that. The model easily switches to different modes of operation and achieves state-of-the-art or close to state-of-the-art performance.

It's quite funny how NLP folks tried to solve low level tasks (POS tagging, NER, Named entity relationship extraction, dependency parsing, sentiment classification etc.) to get to higher level tasks (good summarization, machine translation, text generation, question & answering) and now a single model captures all the low level stuff for free and does high level stuff so good that finetuning it to do low level stuff is unnecessary.

This, the difference between one-shot fine-tuning, vs fine tuning for GPT-2, is one of the major breakthroughs. Since GPT-3 is so hot in the past few days, people seem to forgot or not realize lots of the GPT-3 examples shown off today were possible with GPT-2, with the catch that you had to fine-tune your own GPT-2 model to fit your problem domain (game plots, poems, music, bots that chats like certain characters, etc). GPT-3 makes that fine tuning process unnecessary (although practically you probably can't/can't afford to fine-tune your GPT-3 model)

Are you sure that set out to proof one-shot works? Maybe they found fine-tuning performance disappointing and decided to publish this instead.

> There needs to be a way of absolutely limiting it,

Usually, it's fine-tuning (training a model starting from an earlier model). They can use a small amount of text to fine-tune GPT-3 to their liking.

Well, I've seen examples where GPT-3 strayed quite amusingly. I'm hardly an NN expert, but my understanding that one can't assure the model wouldn't do this (beside retraining on the gamedata corpus alone, which would obviously impair the NN). There are good reasons for game devs to want a measure of certainty here.

Someone else in the thread suggested using a verifier based on game data and maybe that would be fine. The key IMHO must be some kind of NN trained only on game data, either GPT-3 itself or a verifier of some sort.

Why not? It would work for any game set in real past or potential future of today.

There are issues here beyond immersion. One of them is that so far, nobody can peek inside the GPT-3 NN and find out how the game devs trained it.

If a player got something extremely disagreeable from an NPC, was this a fluke or did a dev intentionally add it in training as to make it more likely? There's no way to prove innocence. Add in trigger-happy social media + governments, the potential cost to devs & publishers could get all the way to bans/boycotts/legal threats. Most companies do not wish to risk this, so mitigations must be in place.

Seems like a minor technical challenge (at least in the case of games)

1. Set up a pseudo-adversary NN trained to recognize context-correct speech based on a small corpus. 2. Craft a GPT-3 prompt to get N 9s of accuracy 3.Retry if the answer fails the test from the other NN 4. Set a cap on retries based on how many 9s your prompt got 5. If cap exceeded, return a context-free or limited context response

Sorry. Fixed list:

1. Set up a pseudo-adversary NN trained to recognize context-correct speech based on a small corpus.

2. Craft a GPT-3 prompt to get N 9s of accuracy

3.Retry if the answer fails the test from the other NN

4. Set a cap on retries based on how many 9s your prompt got

5. If cap exceeded, return a context-free or limited context response

This solution would indeed work well enough for me (I'd be paranoid enough to add a tiny blacklist at the end just in case).

In the AI dungeon app, regardless of the setting, you can ask the NPCs to sit down and play super smash brothers with you, and they will.

In case you didnt know, aidungeon.io uses GPT-3 in their "Dragon" model. It's only available for pro accounts though.

I think this might be the way to go forward regarding language models in games: Offer them through cloud computing for GaaS.

This is an excellent example of the potential.

It feels like what I imagined playing in a Star Trek Holodeck would be (well the dialog anyway).

"Give humans a new tool and they will amaze you." You are right. Already some special use cases of the API have come up. I have been exploring the API since yesterday and it is a big deal. But not for the reason of AI and AGI. That may happen later. But right now, the API is the game changer. Single API to apply for lots of different tasks. I have been updating my thread of demos which I am building at https://twitter.com/nutanc/status/1285128265519083520

How long did it take you to get access to the API? I filled out the form almost a week ago and haven't received anything back.

Around 2 weeks I think

That would be a great use. It really takes me out of a game when I talk to an NPC and they run out of dialog options. After that, every time you approach them, they repeat the last thing they said.

I predict it wouldn't be as fun as it might initially sound. NPC dialog is part of level design just like all the rest of the level it occurs on. With few exceptions (such as roguelikes), random level generation tends to produce bland cookie-cutter levels; I don't see why dialog would be any different. The reason it's fun to talk to NPCs in a game like Chrono Trigger or FF7 is because someone put the work in to make their dialog interesting and relevant and fun.

Where I could see it working better would be for e.g. newspaper headlines in a grand-strategy or SimCity-like game. When you do something crazy like have Lichtenstein conquer Western Europe, it could be funny to have some auto-generated commentary on said crazy state of affairs.

It offers you procedurally generated dialogue that can be relevant to any potential player actions. You decide to kill the town chicken and now all the townspeople are talking about it. You've freed up devs from having to hard code all these possible interactions, obviously not perfect but it allows for some interesting possibilities.

Just imagine a caves of qud weirdly coherent storytelling be the base of new games, I LOVE THE FUTURE

>But then I thought of possible applications

Indeed, some pretty impressive demos have been sprouting up on Twitter. Things like generating properly running code from english descriptions of the desired functionality are potentially game changing.

GPT-3 doesn't know how to code. However, it has so many parameters that it was almost able to memorize its training data, which included people asking how to code something and other people answering.

I see people saying this over and over again, with nothing to back it up. I could say the same about the human mind.

And at least those can be run on prem. This thing we are talking about is already in walled gardens.

Even if it was completely rote memorization it is still extremely valuable to be able to give a plain text prompt and get relevant answers back. As SWEs most of us probably have very high end google-fu so we can find potentially obscure answers to our questions but this could make that kind of skill redundant. Why try to search stackoverflow when GPT could just generate exactly the code snippet you need?

Another potential application/adaptation that would be useful is lossy text compression, but I'm not excited about using 300 GB of RAM or a web service to compress and decompress text.

Especially since text is so small anyway, compared to pictures and videos or sound.

It would be great as a "lorem ipsum" generator ...

Reply on news.ycombinator.com