Web LLM
mlc.ai
mlc.ai
"This opens up a lot of fun opportunities to build AI assistants for everyone"
"We have tested it on windows and mac, you will need a gpu with about 6.4G memory."
Although I think Apple doesn't sell computers with less than 8GB RAM, which is used by the GPU as well as the memory is unified.
And latest Valve survey says that at least ~54% of people using Steam who participate in the survey, has a GPU with 8GB or more.
According to https://web3dsurvey.com/webgpu WebGPU enjoys a whopping ~6% browser support.
But the previous two segments are very small ones, if you compare to everywhere computers are being used. Coupled together with that WebGPU is hardly enabled anywhere, I wouldn't be surprised if maybe only ~1% of everyone using their own computer have access to 8GB VRAM + can use WebGPU.
So yeah, bit weird to say everyone when looking at the requirements of 8GB VRAM + WebGPU enabled browser.
That's actually not too bad considering that WebGPU hasn't been released yet (the number will drastically go up when Chrome 113 is out of beta)
Which coincidentally is happening in about 5 hours (but it'll take many days for the update to actually go everywhere) according to https://chromestatus.com/roadmap, exciting :)
I guess privacy and ease-of-use?
So you don't need to download llama.cpp, and run it in the terminal or something?
Actually, given dropbox's deterioration over time, FTP+SVN is sounding pretty good to me right now.
Its also fairly easy to route a Flask server to these models with websockets, so with that I've been able to run python and pass data to the model to run on the GPU and pass the response back to the program. Again, there's probably a better way but its cool to have my own personal API for a LLM.