ggerganov

Just make the same Javscript that provides random moves, make it play 100 games and take the average score.

Then do the same using GPT and compare the scores.

Anything else is just cherry-picking

I just played 2 games randomly pressing all arrow keys with closed eyes and got a score of ~1100 in the first game and 1468 in the second game. OP's AI agent scored 1348

Cool demo

Just want to point out that we are actively working on improving the quantization accuracy / performance of ggml [0]. We already have promising results that are yet to be ported to the WASM code path, so hopefully such type of demos will become a bit better in the future.

[0] https://github.com/users/ggerganov/projects/2

Heh, things seem to be moving in this direction, but I think it's still a very long way to go. But who knows - the amount of contributions to the project keep growing. I guess when we have a solid foundation for LLM inference we can think about supporting SD as well

Hey liuliu, would love if you join the project when you find the time - your work is really inspiring!

I expect LibNC will be better in every aspect: performance, accuracy, determinism. But hopefully with time we will close the gap.

Very inspiring stuff! I hope one day we get to see the magic behind libnc.

"Clean Code, Horrible Performance" :)

Our investigations indicate that it might not be possible to achieve ANE performance improvement over CPU for LLM Decoder inference with batch size of 1 [0]. Just to make it clear - I'm no expert in Core ML / ANE, so these conclusions could be totally wrong.

[0] https://github.com/ggerganov/whisper.cpp/discussions/548#dis...

I haven't run the original PyTorch model not a single time!

I just look at the code and port it. I don't have the hardware to run it.

It currently uses only the CPU via ARM NEON intrinsics - no GPU, no ANE and no Apple Accelerate.

Plan is in the future to utilize respective SIMD intrinsics for other architectures (AVX, WASM SIMD, etc) and also add other more accurate quantization approaches. It's actually not a lot of work and I have most of the stuff ready, so hopefully soon!

Edit: AVX2 support has just been added

Recently I made some progress on efficiently detecting short voice commands (wake words) on RPi4 [0]. Checkout the "command" example in whisper.cpp and it's "Guided mode" operation. There are additional improvements on the way too.

[0] https://twitter.com/ggerganov/status/1602759833312456704

Yes - more of this please :)

Tangentially related and discussed in the past on HN: File transfer via color barcodes and a phone camera

[0] https://news.ycombinator.com/item?id=25459501

[1] https://github.com/sz3/libcimbar

[2] https://cimbar.org

In 1991, the Bulgarian national television aired a prank that the nuclear plant in the country has exploded [0]. Initially, a lot of people were very scared and believed that it really happened.

[0] https://www.nytimes.com/1991/12/23/world/nuclear-prank-in-bu...

Great project! I used it to create a Twitter bot that posts Doom videos (@tweet2doom). Back then it didn't have sound support, but it is great to see that it supports it now.

Edit: I just realised it uses the modifications that I made for my bot to support SDL audio :) Funny thing is, I took these from some other Doom project - ah, the joys of open-source!

There is an attempt for real-time transcription from the microphone at:

https://whisper.ggerganov.com/stream/

It chunks the input in 5 second buffers and processes them independently. Results and performance are not great, but I think good enough for a proof-of-concept.

I was recently playing with the GPT-2 and GPT-J models. Results are often non-sensical for any practical purposes, but I think can be used for making something fun - similar to your IRC bot idea.

If you are interested in running these models yourself without having a beefy GPU, you can try my custom inference implementation. It's in pure C/C++ without any 3rd party dependencies, runs straight on the CPU and builds very easily. I think it is relatively well optimised. For example, on a MacBook M1 Pro I can run GPT-2 XL (1.5B params) at 42ms/token and GPT-J / GPT-JT (6B params) at 125ms/token.

Here are a couple of generated examples using GPT-J:

https://github.com/ggerganov/ggml/tree/master/examples/gpt-j

These are examples using zero-shot prompt where the model auto-completes a text given a starting prompt. You can try to make a conversation bot with a few-shot prompt, but it's not great. Probably the model needs some fine-tuning for that to become feasible.

About a month ago, I made a toy bot that listens to your voice with OpenAI Whisper, generates a response with GPT-2 and vocalizes the response using the Eleven Labs. The TTS quality produced by the Eleven Labs algorithm was mind-blowing to me. The API that they provided was super easy to use. Good product!

Here is a prototype of this that I hacked recently for iPhone:

https://twitter.com/ggerganov/status/1605322535930941441

Thanks for that - I have missed the block-sparse extension of the algorithm when I first read about it. And indeed this seems to be what the author means.

> There has also been a wide variety of accuracy-degrading performance optimizations like Xformers and Flash Attention, which are great tools if you are open to trading accuracy for performance ..

I wasn't aware that Flash Attention trades accuracy for performance. Either I have a wrong understanding of what FA is, or this statement is not fully accurate.

Either way - looks like great work

A follow up on this - I came up with an interesting strategy to achieve this. Still a prototype, but I think it looks very promising:

https://github.com/ggerganov/whisper.cpp/pull/271

The source code is in the `command.cpp` and I will soon write some more details how it works. If you give it a try, definitely let me know if it worked for you.

Hi, it's not obvious how to achieve this, but it feels it could be done. I think all the "tools" are available in the existing interface in `whisper.h` - for example, `whisper_get_probs()` gives you the probability for each token.

This is the smallest GPT-2 model so it usually generates gibberish. Maybe some better prompting could improve the results.

Currently, the strategy is to simply prepend 8 lines of text (prompt/context) and keep appending every new transcribed line at the end:

https://github.com/ggerganov/whisper.cpp/blob/master/example...

Hey author here - I implemented `ggml` as a learning exercise. It allows me to easily port it to WebAssembly or iOS for example.

Btw, there is the `yt-wsp.sh` helper script to download, convert and transcribe a video by given url:

  ./examples/yt-wsp.sh <video-url>

Large v2 has already been added to `whisper.cpp` (check the pinned issues). I was thinking about adding a roadmap soon

I have only implemented the Greedy decoder which is worse compared to the BeamSearch decoder in most cases.

If you have a good GPU then you don't need `whisper.cpp`. Best use case currently is if you are running on Apple Silicon.

I think PyTorch will very soon catch up in performance once Apple Silicon gets properly supported. On x86, I don't observe very big performance improvement compared to current PyTorch. So overall, not sure if it is very worth it, although it's not hard to wrap it thanks to the C-style API.