ggerganov

Correct, I had loaded randomly the "Italian" voice of the Web Speech API.

I did some experiments with adding Metal Performance Shaders support, but the performance that I achieved was only marginally better compared to the one I get when using just the Accelerate framework (there is an unmerged PR with the tests).

Honestly, I am bit confused with all the different types of processing units available on Apple Silicon. If I understand correctly, we have: CPU, GPU, AMX coprocessor and Neural Engine on a single chip. I don't fully understand how these interact with each other. Can we use them all at the same time, or would there be some penalties? I'm interested in finding some resources/information on the topic.

I don't have other recommendations as I am a novice myself when it comes to SIMD. I think the multiplication routines in `whisper.cpp` are relatively basic - dot product and fused multiply-add. With a few trial and errors I came up with these implementations - not sure if they are optimal.

This is my favorite example! I recorded a few video demonstrations where I talk with the AI, but they all sounded very cringe. So, the flagship video demo of talk.wasm is currently of 2 browsers talking with each other, which I think is not as impressive. If somebody gets this running and manages to record a conversation - would be happy to hear it!

I am not very good with Python, so there is some chance I am doing something wrong. But my explanation of the results that I get is that PyTorch currently does not fully utilize FP16 + Metal or AMX when running on Apple Silicon. In contrast, my implementation stores the weights in 16-bit floating point precision (FP16) and also utilizes the AMX coprocessor through the Accelerate framework. As I mentioned in OP, the latter is very efficient for doing the matrix multiplications. According to my experiments, it is comparable in performance to running them on the Apple GPU via Metal.

Thanks! I'm also interested in seeing a CPU comparison against tinygrad. From what I've seen, tinygrad already utilizes the AMX coprocessor, so I expect to have comparable performance between the 2 on Apple Silicon.

Thanks! Indeed, I agree that the project has an educational aspect and value. For me, it helped me get a better understanding of the neural network layers involved in the transformer model. Also, it was a good playground to practice my low-level optimization techniques. I guess another cool thing was that with the help of the community, we came up with a faster way to evaluate the Encoder (at the cost of some accuracy), which ultimately enabled the WASM and RPi4 examples (see #137 if interested in the discussion).

whisper.cpp provides similar functionality via the `livestream.sh` script that performs transcription of a remote stream [0]. For example, you can transcribe BBC radio in 10s chunks like this:

  $ make base.en
  $ ./examples/livestream.sh http://a.files.bbci.co.uk/media/live/manifesto/audio/simulcast/hls/nonuk/sbr_low/ak/bbc_world_service.m3u8 10

  [+] Transcribing stream with model 'base.en', step_s 10 (press Ctrl+C to stop):

  Buffering audio. Please wait...

   here at the BBC in London. This is Gordon Brown, a former British Prime Minister, who since 2012 has been UN Special Envoy-
   Lemboy for global education. We were speaking just after he'd issued a rallying cry on the eve of the 2022 Football World Cup. For
   Governments around the world to pressure Afghanistan to let girls go to school. What human rights abuses are what are being discussed as we...
   run up and start and have happened and have seen the World Cup matches begin. And it's important to draw attention to one human rights abuse.
   that everyone and that includes Qatar, the UAE, the Islamic organization of countries, the Gulf Corps...

[0] https://github.com/ggerganov/whisper.cpp/blob/master/example...

I was thinking, what part of HN comments do you think are already AI-generated?

As a human, I cannot give an accurate estimate. /joke

Hey, here is a frog that can talk! Ask it anything!

- Human: Ok frog, how much is 1 + 1?

- Frog: The answer to your question is 3.

- Human: Meh, wrong answer. Not impressed ..

Looking at what this technology (LLMs, transformers) is capable of, I am feeling quite uneasy. I mean, this is the holy grail of computing that we have always aimed for - an actual general purpose algorithm. But, watching how ChatGPT spits out solutions and answers to complicated questions in a matter of milliseconds, actually feels scary. Yes, it is amazing - but also scary. And this is just the beginning ..

It's one of the reasons I recently ported the Whisper model to plain C/C++. You just clone the repo, run `make [model]` and you are ready to go. No Python, no frameworks, no packages - plain and simple.

https://github.com/ggerganov/whisper.cpp

I really liked the informative and straight-to-the-point about page - describing how the algorithm works in a way that is easy to understand. All the important details are summarised there. Well done!

Edit: From the "How to avoid .." page, there is the following sentence:

> Also, most authorship identification algorithms have poor accuracy when working with small amounts of words. This means the optimal strategy would be discarding an account either after every comment or after a small number of comments. Unfortunately, this is against HN rules and may result in a ban.

Can you clarify what this means and why it would result in a ban?

Ha, this is similar to an idea that I had some time ago:

https://the-story.ggerganov.com

This is again a type of collaborative story-telling where you can vote for each word once per day. New words are added as more votes are accumulated. I thought it would be interesting to see it grow, but unfortunately - it has been just me and a friend submitting votes from time to time :D

Edit: punctuation is intentionally not supported

I agree, but I think things will likely change soon. The main thing holding progress in this field seems to be privacy issues and sending data to the cloud. However, we already have human-level voice transcription that you can run on a low-to-moderate hardware at home, so all it takes is for the average developer to be able to run GPT-like LLMs on their machine. At this point I think the quality will improve really fast.

Ideally it shouldn't. But at the moment, I cannot use Apple Silicon on anything else other than Mac OS and I believe the performance of these chips is unmatched and it will not be anytime soon.

At work I use Gnome and at home I use MacOS - I am used to both environments. I am a C++ developer and spend most of my time in a terminal. Of course, it took some time over the years to have everything customized for my needs, but at the moment I feel equally productive on both.

However, if I had to choose one, I will likely stick with Mac though. The performance of the new Apple Silicon chips is amazing and it's going to get even better.

In my experience, OpenAI's Whisper speech recognition is beyond anything currently out there. Likely Github will use it on the backend.

Very interesting - I was sort of expecting it to happen soon.

I have been playing with using Whisper + Github Copilot in Vim [0]. The Whisper text transcription runs offline with a custom C/C++ inference and I use Copilot through the copilot.nvim plugin for Neovim. The results were very satisfying.

Edit: And just in case there is interest in this, the code is available [1]. It would be very awesome if someone helps to wrap this functionality in a proper Vim plugin.

[0] https://youtu.be/3flN9kTcZJY

[1] https://github.com/ggerganov/whisper.cpp/tree/master/example...

You might find my inference implementation of Whisper useful [0]. It has a C-style API that allows for easy integration in other projects and you can control how many CPU threads to be used during the processing.

[0] https://github.com/ggerganov/whisper.cpp

Plugging here my "text-adventure" mode of Doom that runs on.. wait for it.. Twitter [0] :-)

[0] https://twitter.com/tweet2doom

On M1 Pro, with Greedy decoder and medium model, I can transcribe 1 hour audio in just 10 minutes (~x6 real-time) [0].

[0] https://github.com/ggerganov/whisper.cpp

Me too - I recently ported the model to plain C/C++ and I am now planning to run it on device and see if the performance is any good. Will post an update when/if it works out

Similar project: https://lexman.rocks

I have a vimrc configuration that is compatible with neovim. This way I can use both editors depending on what is available in the current environment. Additionally, there are things that work better with vim and there are things that work better with neovim, so this way I can easily switch depending on what I need.

The problem now, is that it looks like neovim is migrating to lua configuration. For example, this plugin requires a lua config file. This is a bit unfortunate, because I don't see how I am going to maintain a parallel lua configuration for nvim.

You can try my C/C++ port of Whisper:

https://github.com/ggerganov/whisper.cpp

No dependencies, no Python, runs efficiently on the CPU.

According to one of the authors, the code and the models will be available soon [0]

[0] - https://twitter.com/FelixKreuk/status/1575846953333579776

Recently, I've been implementing my custom inference code in C for various models (GPT, Whisper) and am interested to see how it compares to various GPUs in terms of performance. So far, I've been running it only on my MacBook M1 as I don't have the necessary hardware.

Thank you - much appreciated :)

Hi, this is an implementation from scratch - the goal was to become more familiar with how the transformer model works. Also, I implemented inference of the GPT-J model which is quite similar to GPT-2 - it's another example in the same repository.