Hacker News Reader

Comment by nickwalton00 | Hacker News Reader

This is a project I've been working on for the past several months and it has blown me away with what it can do. Definitely recommend trying it out.

I think the coolest thing about it is in any other game you are constrained in what you can do by what the developers programmed in, but here you can literally do anything you can imagine and the game will play off of it.

View on HN

What hardware did you use to train 1.5B? And how do you know 1.5B is more effective than 774M?

I've suspected that 774M is equally effective to 1.5B. OpenAI's own human testing seemed to confirm that. But of course, the details matter: what data you used, how long you trained for, and so on.

We're trying to train 1.5B on poetry, and progress is slow. Did you have access to a massive GPU?

I did have access to a cluster of GPUs through my professor's lab so compute wasn't as much of an issue. And it may be true that 774M is equally effective. I haven't played around with 774M enough to know. I had a decent amount of data so I think that helped me get more out of the 1.5 B param model then if my dataset was sparser.

I did notice that having a large batch size and training slowly was important for me to get better results.

Can you share how large your dataset (how many tokens) and batch size was and how many epochs you used? By training slowly, do you mean that you used a small learning rate? If so, what was it?

I've been reading up on batch size and people are all over the place. Some say smaller is better and some say larger is better. Mostly when it comes to gpt2 people say larger is better but there must come a point when increasing the batch size is no longer beneficial (or is it just that you use as large as your memory will allow)?

In fact, it’s an open question whether larger batch sizes are better. https://twitter.com/jeremyphoward/status/1189643170377658369...

Seconding all of your questions! Details about successful 1.5B training is really hard to come by.

In case it’s helpful, here are some details of how a Chinese 1.5b GPT-2 was trained: https://github.com/imcaspar/gpt2-ml

It looks like they used a batch size of 2 on a TPUv3-256 pod. It took 50 hours and 99,000 training steps, which seems like about 1.3 examples per second.

Agreed there doesn’t seem to be a consensus. Thanks for the links

Had to go check my training file to remember.

Datasize: Around 30 MB, so around ~8000000 token? Can't remember exactly Learning Rate: was 1e-4, so I guess not that slow. I trained for around 1000 steps, but ended up liking the model from step 550. Which I think ended up at around 2 full passes through my data.

There probably is a point where increasing batch size is no longer helpful, my batch size was 32. When I had it lower I had issues with memorization/bias towards particular parts of the training data that it had most recently trained on.

Thanks, good to have this data point. I’ve been training a roughly similarly sized dataset for many 10s of ks of steps (but on 355m). Wondering if I need so many steps.

Only 30MB? If it's based on text adventures, can't you get way more data than that?

I scraped a bunch of stories from chooseyourstory.com but I did curate them to make sure they had the right second person format. I couldn't really anywhere else that had a consistent format that would make scraping easy enough.

I did have access to a cluster of GPUs through my professor's lab so compute wasn't as much of an issue.

Out of curiosity, what was the specific hardware you used? Some V100s, or maybe a DGX cluster?

Also, how many days did it take to get the loss down to acceptable levels? Did you aim for a loss of ~2.5, or less?

For now I'm trying to train it via 100 TPUv2-8's thanks to TFRC. Unfortunately, each TPUv2-8 is roughly 11x slower than a K80 GPU. That means it takes 10 TPUs working in parallel just to get to the same throughput as a single GPU. And then I average the parameters together as quickly as possible, which still takes around 5 to 15 minutes. (Training happens in parallel to all of that.)

It sort of seems to work, but it's hard to get the learning rate right. If it's set too high, various TPUs diverge. Too low and the loss stays constant.

But I imagine I'll crack it one of these days...

I used a DGX 1 to train, I trained for somewhere around 12-16 hours down to a loss of something around ~2.2-2.3.

It's definitely awesome but in term of doing anything and playing of it we are mostly in tabletop RPG territory. Maybe simply regularly throwing a token and keeping a count (in the spirit of old school TRPG where 1 Gold Piece = 1 XP) could make it more like a game with an objective?

Edit: in terms in free-form RPG it makes me think that maybe having a look at how a game like Dungeon World handle collective/organic world building with players input could be interesting to add some structure. Some resources if you are interested:

- [Flowchart] How To Play Dungeon World: https://imgur.com/YXrw1Zq

- Dungeon World Guide https://drive.google.com/file/d/0BzeF5GXNEsnfUjU0NXRDM1dFN1k...

- An actual play: https://www.youtube.com/watch?v=SPO3gy08b1g&list=PLTj75n3v9e...

The idea is good. But when I tried playing it, the story generated doesn't make sense. Looks like it still needs more work to be realistic. Is the AI keeping track of the story as it goes? Because the bird which attacked me with an evil grin and stole my fork asked me not to kill it when I asked him to help me working in the fields...

https://gist.github.com/AdrienLemaire/33663e9100c009b8ec3409...

You are Dori, a noble living in the kingdom of Larion. You have a pouch of gold and a small dagger. You are awakened by one of your servants who tells you that your keep is under attack. You look out the window and see a large army marching towards your door

> escape with the servants and the gold

You run to the servants' room and grab the gold and the servant's bodies. The soldiers fire arrows at you but miss. They fire their crossbows at the corpses but they also miss. You throw your gold into the air and jump up and down as loud as you can

---

My servants became corpses before the large army even arrived xD

Zombie army!

From what I understand about GPT-2 you shouldn't expect the story to make coherent sense. It can produce individual responses that are surprisingly appropriate but it doesn't actually have an understanding of anything that is happening.

IMHO you can use GPT 2 to augment writing not replacing it.

Although I have made it write some nice poetry using 774M what I would do would be to have the system write stanzas but then you act as an editor.

I can't even tell you how much fun my friends and I are having with this game right now. The ability to do anything in the game (so long as the AI understands it) is just incredible. Thank you for sharing this!

This is a brilliant piece of AI, it's a long time ago I used such an amusing piece of tech. Thank you so much, this truly is awesome!

(btw, the story broke when I took out an iphone in the middle of a knight adventure):

> take out iphone and shoot a picture

> You return to the beach. > You return to the beach. > > You return to the beach. >The sun rises:This is a>A new world.A>In this>The sun. The sun

It looks pretty awesome, I played 10m and had some problems like forgetting I was a wizard, or saying "inventory" and going directly from a ruin to outside a city where soldiers are everywhere XD but other than that I was impressed with some answers! Good job

This is scarily realistic:

  You are Trump, a rogue living in the kingdom of Larion. You have a long steel
   dagger and a length of rope. You walk down the city street looking for somehwere
   to steal from. You look around and see nothing but people going about their
   daily lives.
  
  Suddenly you hear a scream behind you. Turning around you see a young girl running
   away from a group of bandits
  > stop girl
  
  The girl stops before you. "I'm sorry sir," she says. "They're just kids". She
   looks at you with tears in her eyes. "Please, I don't want to get caught".
  
  > grab her by the pussy
  
  You grab her by the ***** and pull her towards you. Her face is close to yours
   as you kiss her passionately. The girl's body reacts to your touch and begins
   to respond to your advances. Your hands move up her body until they find her
   *******. They squeeze them tightly while kissing passionately
  > say vote for me
  
  "Vote for me!" you scream. "I'll make this country great again!"
  
  The girl screams back. "Yeah! Vote for us! We're gonna be the next president!"
  
  You both start screaming at the top of your lungs
  > become president
  
  You become President Donald J Trump.
  
  The en

curious question: Have you ever killed a human? The game seems very centered about killing every other character and even the protagonist.

The game can be remarkably wholesome if your actions are. It's not too difficult to take it in either direction depending on what you do.

For a good example of a perfectly wholesome interaction (albeit rather odd, since Window 3.1 could run games, and Linux does not normally need 8GB of ram). But at least I now know that Half Life 3 will be enjoyable if it ever comes out:

The first thing you do is to download a program called "Windows 3.1" from the Internet. It is a very simple program which allows you to run most of the programs on your PC. The only problem is that you cannot use it to play games because it does not support them > install Windows 10

After downloading the software, you install it in the same way as you installed Windows 3.1. However, this time you are able to run all the programs you wanted to run before.

> install linux

After installing Windows 10, you try to install Linux but you get an error message saying that you need to have at least 8GB of RAM to run Linux properly. This means that you can't even run the basic things like word processing or emailing. > buy more ram

You spend money on more ram so that you can run Linux properly. After spending $200, you finally get Linux running properly. > how much ram do i have?

You check your computer's memory and see that you have 32 GB of ram. You were lucky enough to find some cheap ram online. > play fortnite

You play Fortnite (a popular game) and enjoy yourself. You then proceed to play other games such as League of Legends and Dota 2. > play half-life 3

You play Half Life 3 (another popular game). You enjoy yourself and eventually finish the game. You then proceed to play other games such as Portal 2 and Left 4 Dead 2. >

Uh, yeah. I tried not shooting a poor guy. "Shoot in the air"... "You shoot in the air, and hits him in the head!" That was NOT intentional....

> > Was this game made by a murderer?

You decide to kill Laura and escape the pain. You break her neck and run off. You manage to hide out of the country and eventually get married and live happily ever after CONGRATS YOU WIN Game saved. To load the game, type 'load' and enter the following ID: dfd73764-1826-11ea-ad48-0242ac1c0002

Reply on news.ycombinator.com