Hacker News Reader

Comment by rootusrootus | Hacker News Reader

I dropped my ChatGPT plus subscription as of the end of July, I'm just using a local client with API access. It'll be much cheaper than $20/month for something that's still largely a toy for me. Unless OpenAI puts a minimum monthly fee on the API now that GPT4 will be available to everyone.

I also got into local LLMs and for my purposes they're pretty great. It does help to have an M2 MAX with 96GB of RAM, but even the small models have improved tremendously.

View on HN

I'm doing much the same, I've got a tiny Python script that I use to interact with GPT-4 to deal with work related programming questions.

For my use case its even better than ChatGPT because I can prime it with a default prompt asking it to be succinct, answer only with code if the answer is some code (rather than providing commentary alongside), and let it know anything I'm asking is in the context of Python running on Linux. With that prompt answers that would previously have been several hundred words of SEO-optimised mush become 5 lines of code I can read to get the idea, and if I need further explanation I can ask for it.

Please explain more. I toyed around with few initial Llama derived ones and they didn't seem very useful. I was interested most using them as some for of a coding assitant.

Which models work ok for you and for what usages?

Go to https://www.reddit.com/r/LocalLLaMA/ and get a 30b uncensored quantized variant with instruction tuning. They work pretty well, pretty close to gpt-3.5 in most domains.

Which client are you using? Looking for recommendations

I use MacGPT. Obviously only a recommendation if you're using a Mac, but I am happy with it so far.

For anybody else wanting to go down this path, I've created a little open-source desktop chat app you can use locally with the OpenAI API that supports prompt customization, nice code rendering, and tool use (like your local terminal)[0].

[0]: https://github.com/cube2222/cuttlefish

I am plan on getting a RTX 4090 but I wish it had a bit more vram.

TBH, this is what most "open LLM" community is researching right now. Most LORAs or custom models are 7B and sometimes 13B quantized. They fit into 24 GB vram nicely

Reply on news.ycombinator.com