Hacker News Reader

Comment by bioemerl | Hacker News Reader

I hate seeing these guys succeed because everyone of their successes is a new day that AI becomes less accessible to the average person and more locked behind their APIs.

View on HN

I see this statement a lot and have no idea how people come to this conclusion. I have a beefy 16k$ workstation with 2 4090s and I could barely run the LLAMA 65B model at a very slow pace. Let us say we do have the model weights to GPT-4 and GPT3.5, me as the average consumer I don't know how this helps me in any way. I need to shell at least 25k (possibly much more for GPT-4) before I can run these models for even inference, and even then it will be a slow, unpolished experience.

On the other hand OpenAI’s API makes things blazingly fast and dirt cheap to the average consumer. It honestly does feel like they have enabled the power of AI to be accessible to anyone with a laptop. If that requires fending off competition from Behemoths like Google, Meta by not releasing model weights then so be it. This critique would be more apt to Nvidia who are artificially increasing datacenter GPU prices thus pricing out the average consumer. OpenAI is doing the opposite.

An "average" person is not someone who knows how to call an API. Perhaps only on HN

Then an "average" person is certainly not someone who is able to download and run an LLM on their device.

If they don't know how to call an API, they won't know how to run local models (at the moment it's quite a pain to set-up all the dependencies)

It's actually not bad, the hard part is getting the hardware. Kobold will install itself most of the time with a double click.

"AI" as we know it is hardly 6 months old now, just wait a while and it'll be grandma accessible.

You exaggerate a bit! Machine learning and language models have been around for decades. OpenAI itself has been around since 2015.

> LLAMA 65B model at a very slow pace

How does it compare to GPT 3.5, or 4? I mean if you ask the same questions. Is it usable at all?

I tried the models that work with 4090 and they were completely useless for anything practical (code questions, etc.). Curiosities sure, but on Eliza level.

Is there a simple question / answer that you would find illuminating?

the one that I used for GPT-4 and the local ones was a bit obscure:

"how to configure internal pull-up resistor on PCA9557 from NXP in firmware"

the GPT4 would give a paragraph of

> The PCA9557 from NXP is an 8-bit I/O expander with an I2C interface. This device does not have an internal pull-up resistor on its I/O pins, but it does have software programmable input and output states.

and then write a somewhat meaningful code. the local LLMs failed even at the paragraph stage

could you try that?

> . I need to shell at least 25k (possibly much more for GPT-4) before I can run these models for even inference

Give it a decade and you might be able to, but without the model you'll never have the option.

I have been thinking about trying to do LoRA style fine tuning of Flacon-40b or Falcon-7b on RunPod. The new OpenAI 16k context and functions thinking made me lose the urge to get into that. Was questionable whether it could really write code consistently anyway even if very well fine tuned.

But at least that is something that can be attempted without $25k.

- The API is extremely cheap

- There are plenty of open source tools built on top if it (example list: https://github.com/heartly/awesome-writing-tools)

While I wish this work was open, they are both the best and cheapest option out there... by a mile.

This is exactly my problem. They are doing quite well, and closing the door behind them. Open AI isn't your friend and reserves the right to screw you down the line.

Oh, I 100% agree. I just haven't seen another model come close. I'd love to hear someone tell me why.

Lots and lots of human labor. https://gizmodo.com/chatgpt-openai-ai-contractors-15-dollars...

Thing is, as long as the field is growing in capabilities as fast as it is, there isn't going to be any kind of "democratizing" for an average person, or even average developer. Anything you or me can come up with to do with LLM, some company or startup will do better, and they'll have people working full-time to productize it.

Maybe it's FOMO and depression, but with $dayjob and family, I don't feel like there's any chance of doing anything useful with LLMs. Not when MS is about to integrate GPT-4 with Windows itself. Not when AI models scale superlinearly with amount of money you can throw at them. I mean, it's cool that some LLAMA model can run on a PC, provided it's beefy enough. I can afford one. Joe Random Startup Developing Shitty Fully Integrated SaaS Experience can afford 100 of them, plus an equivalent of 1000 of them in the cloud. Etc.

Yeah, I guess it is FOMO and depression.

Less accessible compared to what?

Reply on news.ycombinator.com