Hacker News Reader

Comment by anhner | Hacker News Reader

I'm hoping of one day running GPT3/ChatGPT on my local computer, similarly to how one can run Stable Diffusion now.

I would love to have a personal conversation with these AI systems, use them as a sort of assistant, without the worry of being spied on. At the moment I can't use it as more than a glorified search engine, because of the privacy implications of running it on the cloud.

View on HN

There will always be this uncomfortable balance between need to know information to be useful being essentially the same as everything an adversary would need to really act against you.

With something like chat gpt being plugged into some voice assistant thing and having access to all your documents, emails, and other content, you could imagine having conversations about work content, content creation, calendar management, etc. Basically it would become like a secretary that is able to write letters based on your input, manage your calendar, etc. It could be pro-active and remind you about things, summarize incoming messages, search through your documents, message history, etc.

That's where AI becomes really useful. But the issue of trust is a big one. I don't think a lot of this requires a lot of breakthroughs either just a lot of integration work and engineering. Chat gpt is more a proof of concept than a well integrated thing at this point. It basically is running in isolation and it's only window to the world is chat. Changing that should not be that hard. Running things locally might help with this but it may not be a hard requirement for this. All depends on how useful this is.

> That's where AI becomes really useful. But the issue of trust is a big one

Well google already has access to all of this data, the difference is unlike google assistant, ChatGPT actually can do something useful.

There will always be an edge to these massively operated central models that you can't easily run at home, due to compute and cost. Things like 4-bit quantization will help making these foundational models significantly easier to operate on smaller hardware, and of course you might not need a foundational model for manby local usecases, but maybe a smaller, more specialized LLM is enough (those can even be trained by the bigger models).

Heh. Computers used fill an entire room, and cost the equivalent of a house. And now everyone carries one way more powerful in their pockets for the cost of a few meals in a restaurant.

And yet we have supercomputers that dwarf any phone.

There's not much of an alternative: even if compute power gets extraordinarily cheap / models are very optimized, either you have a very, very large hard drive, or you have to use the internet for that. You just can't hope to have a model that is trained to know everything in a smallish file, the weights need to be at least as heavy as an ideally compressed version of all the things it knows (which is of course much less than the huge amount of data it is trained on, mainly due to redundancy, but still a lot)

Of course, right now you also need at least 8 super-expensive A100 GPUs and not just your laptop CPU, but maybe that's going to change eventually.

Does it need all that much space though? I mean Stable Diffusion was trained on hundreds of terrabytes of images, and the model only needs 4-5 GB of hardware space and a decent GPU to run.

I haven't read anywhere any stats about GPT3/ChatGPT yet (like how big the model is)

These models are way too big for consumer hardware.

Is it bigger than the Stable Diffusion model? (4-5 GB)

Reply on news.ycombinator.com