Comment by sashank_1509 | Hacker News Reader

Comment by sashank_1509 | Hacker News Reader

Parent Full thread

I see this statement a lot and have no idea how people come to this conclusion. I have a beefy 16k$ workstation with 2 4090s and I could barely run the LLAMA 65B model at a very slow pace. Let us say we do have the model weights to GPT-4 and GPT3.5, me as the average consumer I don't know how this helps me in any way. I need to shell at least 25k (possibly much more for GPT-4) before I can run these models for even inference, and even then it will be a slow, unpolished experience.

On the other hand OpenAI’s API makes things blazingly fast and dirt cheap to the average consumer. It honestly does feel like they have enabled the power of AI to be accessible to anyone with a laptop. If that requires fending off competition from Behemoths like Google, Meta by not releasing model weights then so be it. This critique would be more apt to Nvidia who are artificially increasing datacenter GPU prices thus pricing out the average consumer. OpenAI is doing the opposite.

> . I need to shell at least 25k (possibly much more for GPT-4) before I can run these models for even inference

Give it a decade and you might be able to, but without the model you'll never have the option.

I have been thinking about trying to do LoRA style fine tuning of Flacon-40b or Falcon-7b on RunPod. The new OpenAI 16k context and functions thinking made me lose the urge to get into that. Was questionable whether it could really write code consistently anyway even if very well fine tuned.

But at least that is something that can be attempted without $25k.

> LLAMA 65B model at a very slow pace

How does it compare to GPT 3.5, or 4? I mean if you ask the same questions. Is it usable at all?

I tried the models that work with 4090 and they were completely useless for anything practical (code questions, etc.). Curiosities sure, but on Eliza level.

Is there a simple question / answer that you would find illuminating?

the one that I used for GPT-4 and the local ones was a bit obscure:

"how to configure internal pull-up resistor on PCA9557 from NXP in firmware"

the GPT4 would give a paragraph of

> The PCA9557 from NXP is an 8-bit I/O expander with an I2C interface. This device does not have an internal pull-up resistor on its I/O pins, but it does have software programmable input and output states.

and then write a somewhat meaningful code. the local LLMs failed even at the paragraph stage

could you try that?

An "average" person is not someone who knows how to call an API. Perhaps only on HN

Then an "average" person is certainly not someone who is able to download and run an LLM on their device.

If they don't know how to call an API, they won't know how to run local models (at the moment it's quite a pain to set-up all the dependencies)

It's actually not bad, the hard part is getting the hardware. Kobold will install itself most of the time with a double click.

"AI" as we know it is hardly 6 months old now, just wait a while and it'll be grandma accessible.

You exaggerate a bit! Machine learning and language models have been around for decades. OpenAI itself has been around since 2015.

Reply on news.ycombinator.com