Hacker News Reader

Comment by davidatbu | Hacker News Reader

I'm curious, why would there be doubts?

GPT-4 is incredibly useful, and it's clear there's so much straightforward bits of additional product development on top of it in various verticals that will be incredible productivity multipliers in the respective verticals.

View on HN

There's a real danger that people will just run LLMs on their local GPUs. It's not a guarantee that an online service is a profitable idea.

Going to need some insanely beefy GPU cluster at your house to run anything close to GPT-4.

The best I can run is either LLaMa 65B at 1 token per second (too slow) on my CPU or LLaMa 30B on my GPU at quite a fast ~30 tokens a second.

Nowhere near the usefulness of GPT-4.

GPUs in the future will get more powerful. Especially if they are designed with LLMs in mind.

I wonder where a critical threshold might lie? For example, right now if you only have 24Gb of VRAM, the best you can run locally at usable speeds are the 4-bit quantized LLaMa 30B models, which are only semi-usable. If you have 48Gb however, you can run the 4-bit quantized LLaMa 65B, which is much better.

I assume there will be advances that basically make the context window practically infinite. That ought to make the lower parameter count models much more powerful on its own. I also assume they will become more efficient to run through sparsity/pruning, though I'm a total novice on this topic.

I wonder how many generations of hardware are we talking? One? Two? Or three? It feels like it's potentially within that range.

You could say the same about many SaaS projects. Why pay for an expensive GPU upfront and then some guy who can install it, configure it, create some sort of interface for you to talk to it... when you can just pay openai to do it for less money?

Because GPU-Servers that can run a typical LLM are less than 5k, which includes installation. Really, running an LLM seems to be no more complicated than running a NAS from a system administrators perspective.

Emphasis on running, not training.

5k can't even get you 80 GB of VRAM on a GPU. How is that possible?

I expect there to be GPT-4 level model (or close enough) that anyone can run in the major cloud providers soon. In this future, most AI powered products won’t be paying OpenAI anything.

Google still catching up for more than a month. Amazon still doesn't have anything comparable, but definitely needs and wants it. So it doesn't looks like it's easy to replicate. Besides, it's not a finished product. It's a service in continuous development. For how long OpenAI will be able to stay ahead of competitors is yet to be seen. They are trying to create a secondary market of resellers. This may keep them in business for long time if they are just "good enough".

It sounds like it is trivial to replicate GPT-4 performance.

I Would be surprised if that is the case.

Not trivial, but not beyond the realm of possibility either.

Cost to use openAI is also pretty low compared to hosting models

But they are losing money.

Advertising? Who's going to click on advertising that doesnt do it now? Subscription? Its useful but how many people will pay a sub for it, its not netflix or porn. Licensing maybe? Incorporating it into other products but the competion is going to be intense so prices will drop or they'll have to be the best.

This isnt going to be the moneymaker people think it will be..

> Subscription? How many people will pay a sub for it?

I'm a programmer, and I pay a subscription for GPT-4 because it's ridiculously worth it.

I'm curious, what do you do that you don't think GPT-4 is worth it?

It might be worth it for you, but how many people like you are there in the market? Not many I suspect. Most wont be able to afford it or will go for free options.

Since we're both not providing data, I'd be interested in hearing your personal experience on this (if it's not too much of a bother, feel free not to respond!):

What do you do for a living, and do you find GPT-4 helpful for it?

I suspect as soon as top management realizes how useful it can be on all levels there will me mass subscriptions from companies. Of course OpenAI needs to convince it's safe and secure. New dark mode will help.

But do they have moat? Or could the existing big tech companies enter the same market and offer the similar enough product? It is not like Amazon, Google, Microsoft and maybe even Facebook don't have enough money to throw at it and sell something.

Reply on news.ycombinator.com