Hacker News Reader

Comment by minimaxir | Hacker News Reader

Having enough scale to perpetually offer free/low-cost compute is a moat. The primary reason ChatGPT went viral in the first place was because it was free, with no restrictions. Back in 2019, GPT-2 1.5B was made freely accessible by a single developer via the TalkToTransformers website, which was the very first time many people talking about AI text generation...then the owner got hit with sticker shock from the GPU compute needed to scale.

AI text generation competitors like Cohere and Anthropic will never be able to compete with Microsoft/Google/Amazon on marginal cost.

View on HN

And ChatGPT has a super low barrier to entry while open source alternatives have a high one.

Creating a service that can compete with it on that regard implies you can scale GPU farms in a cost effective way.

It's not as easy as it sounds.

Meanwhile, openai still improves their product very fast, and unlike google, it's their only one. It's their baby. It has their entire focus.

Since for most consumers, AI == ChatGPT, they have the best market share right now, which mean the most user feedback to improve their product. Which they do at a fast pace.

They also understand that to get mass adoption, they need to censor the AI, like MacDonald and Disney craft their family friendly image. Which irritate every geeks, including me, but make commercially sense.

Plus, despite the fact you can torrent music and watch it with VLC, and that Amazon+Disney are competitors, netflix exists. Having a quality service has value in itself.

I would not count open ai as dead as a lot of people seem to desperately want it to be. Just because Google missed the AI train doesn't mean wishful thinking the market to be killed by FOSS is going to make it so.

As usual with those things it's impossible to know in advance what's going to happen, but odds are not disfavoring chatgpt as much as this article says.

> Having enough scale to perpetually offer free/low-cost compute is a moat.

Its a moat for services, not models, and its only a moat for AI services as long as that compute isn’t hobbled by being used for models which are so inefficient compared to SOTA as to waste the advantage, which underlines why leaning into open source the way this piece urges is in Google’s interests, the same way open source has worked to Google and Amazon’s benefits as service providers in other domains.

(Not so much “the ability to offer free/low-cost compute” but “the advantages of scale and existing need for widely geographically dispersed compute on the cost of both marginal compute and having marginal compute close to the customer where that is relevant”, but those are pretty close to differenly-focussed rephrasings of the same underlying reality.)

That's what a lot of people think until they run Vicuna 13B or equivalent. We're just 5 months in this, there will be many leaps.

Yes there will, that's the problem HN User Minimaxir is talking about.

It will only get less and less expensive for Microsoft in terms of cost. And more and more effective for Microsoft in terms of results delivered.

How do you compete with free? That's the question. The previous internet experience has already shown us that "also be free" is not really a sustainable or even effective answer. You have to be better in some fundamental dimension.

What makes you think open ai won't look at the FOSS improvements, include them in their tech, and make their GPU farm way cheaper, rendering their service even more competitive?

Not to mention it's easy to run stable diffusion, but midjourney is still a good business. I can run sd on my laptop, I still pay for midjourney because it's convenient, the out of the box experience is better than any competition, and it keeps improving.

that's like saying that apple and MS can look into linux and steal ideas. Yes they can do that but it doesnt make linux any less useful. If anything they learned to contribute back to the common pile, because everyone benefits from it. It would be a problem if this was a one-way relationship , which it doesnt seem to be. If Open source is making them more money, why kill it

You are making my point: linux, mac and windows coexist, despite the overwhelming strength of open source, and the proprietary platforms are quite profitable.

But the point is not to kill commercial software because then OSS will die too because people will have to find other jobs

I mean, read the article, the author is concerned about that, and wants Google to open source more so it's not just Facebook's lama that gets open source building on it.

The reason why proprietary software ever had a moat simply comes down to: software startups could dump investment capital onto the development process and achieve results much faster, with better user interfaces, allowing them to achieve path dependence in their customer base. Thus we had a few big application verticals that were ultimately won by MS Office, Adobe Photoshop, etc.

If the result here is as marginal as it seems - a few months of advantage in output quality and a slightly more sleek UI - the capital-intensive play doesn't work. The featuresets that industrial users want the most depend on having more control over the stack, not on UI or output quality. The open source models are stepping up to this goal of "cheap and custom". Casual users can play with the open models without much difficulty either, provided they take a few hours to work through an installation tutorial - UI isn't a major advantage when the whole point is that it's a magic black box.

> Casual users can play with the open models without much difficulty either, provided they take a few hours to work through an installation tutorial

That can be quite a barrier for entry for non-powerusers. I wouldn't underestimate serving casual users, considering that the alternative is OSS i.e. giving your shit away for free.

Charity is only a moat if it’s not profitable.

There's "immediately profitable" and "eventually profitable". Vast compute scale allows collection of customer generated data so the latter is possible, AI as of yet is not the former.

So GP point still stands. FAAMG can run much larger immediate deficits in order to corner the market on the eventual profitability of AI.

The amount of valuable data generated from professionals using these services to work through their problems and find solutions to industry problems is immense. It essentially gives these companies the keys to automating many industries by just...letting people try and make their jobs easier and collecting all data.

All this talk that every investment pays off in the end is faulty and dangerous. Many investments don't pan out, 95% of the firms you see in the ticker this decade might be gone, and yet everyone is very confident is underwriting these "losses for future gains" but really it's economies of scale. It doesn't cost MSFT much more to run the GPU than to turn it on in the first place.

> . FAAMG can run much larger immediate deficits in order to corner the market on the eventual profitability of AI.

This assumes that there is a corner-able market. Previously, the cost of training was the moat. That appears to have been more of a puddle under the gate than an actual moat.

It seems the plan is to be a loss leader until scale is sufficient to reach near AGI levels of capability.

There was some indication recently that OpenAI was spending over $500k/day to keep it running. Not sure how long thats going to last. AGI is still a pipe dream. Sooner or later , they’re going to have to make money.

Assuming you're talking about the free ChatGPT product, it's important to consider the value of the training data that users are giving them.

Beyond that, they are making a lot of money from their enterprise offerings (API products, custom partnerships, etc.) with more to come soon, like ChatGPT for Business.

I know there are use cases out there, so it's not a dig. I'm curious how many enterprises are actually spending money with OpenAI right now to do internal development. Have they released any figures?

Oh no, they're going belly up in 20,000 days! (i.e. $10B / 500k) Compute is going to keep getting cheaper and they're going to keep optimizing it to reduce how much compute it needs. I'm more curious about their next steps rather than how they're going to keep the lights on for ChatGPT.

https://www.youtube.com/watch?v=z9OUZNicTGU&t=123s

$500k/day for a large tech company is absolutely peanuts. Open.AI could probably even get away with justifying $5M/day right now.

In other words, engage in anti-competitive behavior.

This is the timeline that's scaring the shit out of them:

Feb 24, 2023: Meta launches LLaMA, a relatively small, open-source AI model.

March 3, 2023: LLaMA is leaked to the public, spurring rapid innovation.

March 12, 2023: Artem Andreenko runs LLaMA on a Raspberry Pi, inspiring minification efforts.

March 13, 2023: Stanford's Alpaca adds instruction tuning to LLaMA, enabling low-budget fine-tuning.

March 18, 2023: Georgi Gerganov's 4-bit quantization enables LLaMA to run on a MacBook CPU.

March 19, 2023: Vicuna, a 13B model, achieves "parity" with Bard at a $300 training cost.

March 25, 2023: Nomic introduces GPT4All, an ecosystem gathering models like Vicuna at a $100 training cost.

March 28, 2023: Cerebras trains an open-source GPT-3 architecture, making the community independent of LLaMA.

March 28, 2023: LLaMA-Adapter achieves SOTA multimodal ScienceQA with 1.2M learnable parameters.

April 3, 2023: Berkeley's Koala dialogue model rivals ChatGPT in user preference at a $100 training cost.

April 15, 2023: Open Assistant releases an open-source RLHF model and dataset, making alignment more accessible.

this doesn't even include the stuff around agents and/or langchain

The post mentions that they consider "Responsible Release" to be an unsolved hard problem internally. It's possible that they are culturally blind to agents.

They're basically saying that Pandora's Box, assuming it exists, has already been open. Even if OpenAI, Facebook AI Research and Google DeepMind all shut down tomorrow, research capable of producing agents will continue worldwide.

Interesting! It's like nothing has happened on the field for the last three weeks heh

OpenLlaMa came out last week I think.

The doc was written a bit ago.

This really ought to mention https://github.com/oobabooga/text-generation-webui, which was the first popular UI for LLaMA, and remains one for anyone who runs it on GPU. It is also where GPTQ 4-bit quantization was first enabled in a LLaMA-based chatbot; llama.cpp picked it up later.

> AI text generation competitors like Cohere and Anthropic will never be able to compete with Microsoft/Google/Amazon on marginal cost.

Anthropic already does, with its models. They are same price or cheaper than OpenAI, with comparable quality.

> Having enough scale to perpetually offer free/low-cost compute is a moat.

Rather than a moat it is a growth strategy. At one point in time you need to start to monetize and this is the moment when rubber hits the road. If you can survive monetization and continue to grow, now you have a moat.

A good example of this is Youtube

Reply on news.ycombinator.com