WyGPT: Minimal mature GPT model in C++ | Hacker News Reader

WyGPT: Minimal mature GPT model in C++

github.com

65 pointswangyi_fudan15 comments

This project comes from the author of wyhash, which is now the default hashing algorithm in Go, Zig etc.

What's lacking is more information about the project. I was able to compile it and just started training on GPU, but I have no idea what's going on, when it will end, and what the characteristics are in general. Currently the only output I get is:

  ./train book-war-and-peace.txt
  book-war-and-peace.txt 3.05397 1
  para 1679360
  32 1.3654 206.1985

I guess I need to dive into the source code in the meantime :)

2.5 years of effort but the author certainly didn't spend even 2.5 mins writing code for more descriptive outputs. Meanwhile I'm also training in order to find how well it works but I only have a small GPU. I would appreciate if someone could provide a trained model.

So it's a character-based model (makes sense, because there's no tokenizer to be found). The model file is 3.3 MB

Output after 3 hours of training (loss = 0.4160):

  hello, the sounds of the rain assistants. In provations
  which had been banitz had taken step in, the spirit of the crowd, and
  the right winot to scrap on , Valuex, one of marks othhhanled me in a
  most powerful power. The other Augun had been in
  commands among that guests, the regiments were placed by imists, and
  science of irritation, but or thoughts that he would never cease to point
  before meant in the losses were discussing, and at once she had lived
  in the church.

From the looks of it, it's similar to MinGPT/nanoGPT (I found a Reddit thread [0] where the author compares it to MinGPT)

[0] https://www.reddit.com/r/MachineLearning/comments/o2u2cm/pwy...

The readme page is cryptic. What does 'mature' mean in this context? What is the sample text a continuation of?

Having a gif the thing in use would be great, similar to the gpt4all readme page. (https://github.com/nomic-ai/gpt4all)

If you think the readme is cryptic, you're in for a treat

Was pleasantly surprised at how accurate this comment is.

Went to see that code repo...guy threw a party there

sexy is part of Neanderthal. Hope this helps

tried reading the cuda source and I can see why it took 2.5 years to get working properly

Gross. Why in the world do I want a gif for a tiny code release? Just read the source.

This looks extremely interesting, considering the author I'm inclined to give it the benefit of the doubt.

The source looks fugly, but who cares, I can reformat it. Compilers don't care.

True, maybe a gif is a bit of an inane ask for a relase of this sort.

sexy self is a cool naming convention that will possibly replace foobar https://github.com/wangyi-fudan/wyGPT/blob/main/cpu.cpp#L140

I love every ambitious and tiny codebase, but the formatting looks very uncommon and for me personally - hard to read. Is there a tool that produces it?

The code is wizardry. It looks more confusing than obfuscated JavaScript

Very neat.

Reply on news.ycombinator.com