Talk = GPT-2 and Whisper and WASM
github.com
github.com
> whisper: number of tokens: 2, 'Hello?'
> gpt-2: I want to have you on my lap.
Imagine there is a guy that likes watching movies. Which ones would he like most in 2022?
That context persists for a while.
The total data that the page will have to load on startup (probably using Fetch API) is:
- 74 MB for the Whisper tiny.en model
- 240 MB for the GPT-2 small model
- Web Speech API is built-in in modern browsers
Everyday we stray further from god's light :/
Coming from a limited bandwidth contract, I hate when I click a link and it instantly starts downloading a huge file.
Great work OP!
Speech to text has higher chances though, that's an interesting idea, as they already package text to speech too.
I can't really see why not anyway, as more things are in the browser it makes sense to me to integrate the ability to "AI check" your text like a grammar or spell checker to improve your writing along some dimensions that you like.
It's not honest, but in kind of the same way that a spellchecker isn't honest and since it's going to be possible anyway I don't see what extra harm it causes to make it accessible for everyone so that we can both actually see an upside and also begin to recognize that text we read is at this point likely to be at least partially AI generated and potentially factually incorrect.
Even better if things like Firefox reader mode, one of my favorite tools, can also do text summarization. Just imagine the adversarial interaction between a tool designed to generate confident sounding fluff and one to summarize confident sounding fluff. Honestly it seems like a likely inevitable future path.
It may as well be part of the browser where it stands a better chance of keeping people's long term attention on the ease of using these tools. Spammers will be able to do it, fake journalists and such will be able to do it, better if we can do it too so that at least we are aware of the potential abuse.
We may be able to reduce the size without sacrificing performance, but that's an area of active research still.
Or, not.
Racter was commercially released for Mac in December 1985:
Racter strings together words according to "syntax directives", and the illusion of coherence is increased by repeated re-use of text variables. This gives the appearance that Racter can actually have a conversation with the user that makes some sense, unlike Eliza, which just spits back what you type at it. Of course, such a program has not been written to perfection yet, but Racter comes somewhat close.
Since some of the syntactical mistakes that Racter tends to make cannot be avoided, the decision was made to market the game in a humorous vein, which the marketing department at Mindscape dubbed "tongue-in-chip software" and "artificial insanity".
https://www.mobygames.com/game/macintosh/racter
https://www.myabandonware.com/game/racter-4m/play-4m
It's only amazing that chatGPT backed by GPT-3 is the first thing since then to do enough better that everyone is engaged.
I owned that in 1985, and having studied AI/ML previously I've been (and remain something of) an AGI skeptic. But now in 2022, I finally think “this changes everything” ... not because it's AI, but because it's making the application of matching probabilistic patterns across mass knowledge practical and useful for everyday work, particularly as a structured synthesis assistant.
edit: whisper is awesome
A) ggml https://github.com/ggerganov/ggml/tree/master/examples/gpt-2
B) Fabrice Bellard's GPT2C https://bellard.org/libnc/gpt2tc.html
Is it anywhere near being able to be run on local consumer hardware?
How long until we can have the GPT3 or 3.5 chatbot locally like we have StableDiffusion locally for image generation?
I've been spoiled by having it accessible offline and with community built support/modifications to it. GPT-3 is super neat but feels like too many guard rails or the custom playground is too pricey.
For inference so basically running it you need multiple GPUs and hundreds of GBs of GPU memory. As to model size it's around 100x bigger than SD. You can forget about running it locally unless you have dozens of high-end GPUs or you want to wait hours/days/weeks (depending on your hardware) for a single response.
That being said, neither GPT3 nor GPT2 are "efficient" models.
On the one hand, they use inefficient architectures - starting with using a BPE Tokenizer, to having dense attention without any modifications, to being a decoder only architecture etc. Research has come up with many more fancy ideas on how to make all this run better and with less compute. But there is a reason why GPT2/3 are architecturally simple and inefficient: we know how to train these models reliably (more or less) on thousands of GPUs, whereas the same might not be true for more modern and efficient implementations. For instance, when training OPT, Facebook started using more fancy ideas but finally ended up going back to GPT-3 esque basics, simply because training on thousands of machines is a lot harder than it seems in theory.
On the other hand, these models have far too many parameters compared to the data they were trained on. You might say they are undertrained - or they lean heavily on available compute to make up for missing data. In any case, much smaller models (like Chinchilla by DeepMind) match their performance with less parameters (and hence compute or model size) by using more and better data.
In closing, there are better models for edge devices. This includes GPT clones like GPT-J in 8bit, or distilled version thereof. Similarly, there is still a lot of gains that will happen when all the numerous efficiency improvements get implemented in a model that operates at the data/parameter efficiency frontier.
Still, even when considering efficient models like Chinchilla and then even more architecturally efficient versions thereof - we are still talking about a lot of $$$ to train these models. And so we are yet further from having OpenSource implementations of these models than we are from someone (like DeepMind) having them...
With time, you can expect to run coherent models on your edge device. But not quite yet.
At this stage, we are back to improving model efficiency, I think, especially for code models. But not there yet.
Sorry for the rambling, the actual answer is no I do not have a really good codex type model in open source.. yet
(LOVE this demo.)
Currently, the strategy is to simply prepend 8 lines of text (prompt/context) and keep appending every new transcribed line at the end:
https://github.com/ggerganov/whisper.cpp/blob/master/example...