Hacker News Reader

Comment by t-writescode | Hacker News Reader

All this external money raising when it comes to AI scares me.

For companies that are built off these AI services, there's a very real fear that the prices we're getting on API calls aren't covering the costs of the requests themselves under some assumption that costs will magically go down, or we'll get locked in.

I hope the companies like Mistral and OpenAI know that if they're selling API access below cost they could be the origin of a __lot__ of companies closing and choose to make sure their operating costs are sustainable. (I understand creating new features and new models taxes additional capitol and that's a perfectly fine use of VC money. But selling a $1.00 for $0.90 is dangerous and bad.)

View on HN

It's due diligence to understand your cost of goods sold using open source models where possible.

Don't actually spend the business effort to use them if it's not worth it, but have a business continuity plan to be able to switch over, especially if a "single source provider" going out of business or changing it's pricing could represent a existential thread to your business.

It's the same as any other tech vendor.

API prices of the private AI platforms are more or less comparable to self-hosted OSS solutions like Meta's LLAMA 3.

So at least for current gen, you're ok.

TBH, what should scare you is that most of the AI startups are building on top of someone’s APIs.

So, in most cases they have no moat. They are going to close, and while some will close due to operational cost issues as you identified, a lot more will close since they have no defensible business/revenue model.

Right, but one group closes by definitionally not doing anything value added. The other group closes by building something value added within a heavily distorted economic environment.

Agree. But, if the source API pricing is increased, theoretically any startup that has actually created something of value and stickiness should be able to adapt.

Or, even better if they can price future API price increases in their own pricing, today.

There is an interesting dynamic right now. NVidia is selling cards at somewhere around 98% margin, they are then investing money into these startups - who will then spend some fraction back on NVidia hardware. It's a smart strategy by NVidia, as they can effectively leverage past revenue to create future revenue - but it's unclear what business models will be sustainable coming out of this investment.

You can only keep an Ouroboros loop going for only so long. At some point, you'll need to inject fresh, demand-driven, customer revenue into the machine.

Nvidia’s strategy is to build a cloud service to avoid being locked out once all the other major players develop their own chips. So by owning an LLM they could provide their own service with their own hardware.

It’s a classic problem. The trick is to make sure you’re not competing on cost alone. Small to midsized companies are better off focusing on other differentiators. The extra money those differentiators brings makes their business more sustainable, too.

You can actually see it in paid, web stacks. There’s free ones, esp FOSS, that are really good. So, the paid contenders try to be better at other things. There’s many of them in the market, too, despite over a decade of free alternatives.

One thing to consider is that they are EU-based. That means they will meet the EU's regulatory demands (safety, censorship, whatever). They will have a unique place in serving EU customers and look like a safe bet for companies headquartered there. They may also be the default choice for governments in the EU.

Of course, the EU has a large user base and economically strong demographics. Still, even knowing mistral excellence it's hard to conceive that they will serve 27+ countries with different languages, demographics, and cultures.

I'd be interested to know the real cost. Are they selling $1.00 for $0.90 or are they selling $1.00 for $.10?

If it's the latter, way more startups on SOTA models will become unviable once the investor subsidies dry up.

Fear? That's been a fact as far as I've been aware since the big blowup back in mid '23. It got better, ish, with some of the updates they've dropped but it's still catastrophically losing money on every call.

I'm guessing that's going to be the needle that finally pops this bubble, because as neat as a lot of shit going on with various models is, nobody is going to pay what it actually costs to run the things in order to have that neatness.

Citation? Because I haven't been able to find any such thing in my research.

Some quick googling says OpenAI is losing $700,000 per day operating it's models, which some other quick googling say are processing roughly 10 million queries per day, which if we do quick back-of-napkin math is about 7 cents per query.

Currently their lowest-tier price is $20/month, and limits users to 80 queries per hour. If we assume that 7 cents number and our hypothetical user actually uses those 80 queries for whatever odd reason, that's $5.60 it costs OpenAI, which if that user used them for say, 8 hours (the typical work day), that's $44.80 for OpenAI on a user who's only paid $20. And that's one day, that $20 currently gets them a month of access to that 80 queries per hour. I hope for the dream of profitability that they would get throttled at some point. So the user could, in theory, already render their subscription a net loss for OpenAI with 4 workdays of steady usage.

And yes this is strictly back-of-napkin math here but it would explain why "OpenAI" and "bankrupt" keep appearing in the tech press over and over.

> catastrophically losing money on every call

You're going to have to bring some actual numbers on that speculation.

GPU companies providing inference on open source models (like deepinfra or togetherAI) are doing so at an extremely competitive cost, making me think that the API pricing of the big players right now is profitable.

(for example, deepinfra has wizardLM-2-8x22B at $0.65/1M output tokens, compared to $6/1M output tokens for 8x22B by Mistral - and of course Mistral has some more expensive, closed source models that perform better)

Mistral's open models are not large. If Mistral were to shut down users could easily find another provider to run inference or bring it in house. I think even their commercial models are relatively cheap to run (at least compared to flagships from the big houses), so even if Mistral were to increase the usage prices to reflect true costs it is unlikely to bankrupt most users. My 2c.

Why’s that bad? If companies were building something that can be trivialized so easily that’s on them.

Imo. A company can use whatever wrapper services but the business they serve should be smarter.

Why does it scare you? Why do you think it's dangerous and bad?

It's a well-established, economically rational business model to subsidize prices in order to gain market share.

Uber has been very successful at it. Amazon has had great success. It's standard.

Investors know what they're doing and know that some companies won't be able to sustain it and will go out of business. It's a calculated, known risk.

And users don't really need to be concerned. Just enjoy the low prices while they last. If the service you use goes out of business, it's pretty easy to switch to another.

> Why does it scare you? Why do you think it's dangerous and bad?

> It's a well-established, economically rational business model to subsidize prices in order to gain market share.

Because once they have the market share, they raise prices and/or make the product worse in order to get their money back.

> And users don't really need to be concerned. Just enjoy the low prices. If the service you use goes out of business, it's pretty easy to switch to another.

Well that's the question, isn't it? If the market stays healthy and they don't take the oxygen out of the room and kill the competition and switching costs stay low, then yes by all means enjoy the free ride but plan for its end. But... do plan for it.

>they raise prices and/or make the product worse in order to get their money back

They should only be able to do that if they have a considerable moat though?

I think the bigger risk is that they don't get the market share and then they're tempted to make unethical decisions to keep afloat as their unbalanced inputs & outputs start to drive increasing pressure from their investors.

> It's a well-established, economically rational business model to subsidize prices in order to gain market share.

Right but it means you can't depend on that price remaining low. If it's below cost, the price must rise. That may be important for anyone building businesses on top.

> it's pretty easy to switch to another.

Well that assumes the others have true lower prices.

> the price must rise. That may be important for anyone building businesses on top.

Well yes but every business owner knows that, or should know that. That's a known business risk. We're all adults here.

If your business fails because you naively assumed that prices you rely on will never go up, then that's on you.

Every business owner does not know what prices for other companies actually are -- they assume that they are being charged a price which has been determined by the market and is worthwhile to the seller to cover costs and profit.

By taking outside money and subsidizing certain market players so that they can sell their goods at below market value (because that is what is happening) you are undermining the market and punishing players who are acting rationally.

This would be absolutely no different than taking tax money and subsidizing certain companies over others to destroy competition and then claiming that it was a market decision.

'We are all adults here' is not a magical phrase that makes cheating OK.

> Well yes but every business owner knows that, or should know that.

Know that they are buying services below cost? I don't think so.

It's a reasonable bet that compute costs generally trend downwards, and in a competitive market where the key thing you're paying for is compute that should also trend downwards.

That's different to finding out that everyone's been selling things way under the true cost.

> Know that they are buying services below cost? I don't think so.

To the contrary, of course you should know.

If you know anything about tech businesses, you know that startups being funded by massive amounts of VC keep prices artificially low initially, and that they then rise.

This is tech startup 101 stuff.

It's fine if you're not aware of this as an engineer. But if you're building a business and therefore have a business model where you've identified known risks, there's no excuse for you not to be aware of it.

I know that it happens, but that's different from knowing of it's happening here or not, and by how much.

Is openai selling below cost? Is the cost on azure lower than the true cost? ChatGPT free is obviously run subsidized but are the apis?

Can you tell me the actual cost to Microsoft of a gpt4 API call?

Investors don't know what they're doing. This is laughably apparent. They're monkeys throwing bananas at dart boards. There's that whole negative externality thing. There aren't alternatives to FAANG because they've viciously used their VC money to drive out competition, used their lawyers to litigate competition into extinction, and every other tool at their disposal to press an advantage in consumer exposure. This isn't a fair market where the best product wins; it's a game of "who's got the most money."

https://en.wikipedia.org/wiki/Enshittification

This is how the world gets shittier. It's a whole thing. It's a large part of why things are as bad as they are.

But hey, yeah, don't worry about it, they obviously know what they're doing.

Because they are platform companies.

If I launch a business that makes sense given the current pricing of LLMs, and the price then goes up by a factor of ten, my own unit economics may not make sense any more.

There's an Uber comparison here: back when Uber was cheap, some people made life decisions about where they lived and worked that were predicated on being able to afford a commute in an Uber (since public transport doesn't serve a lot of places, at least in the US).

This hurt a lot when the prices went up a few years later!

Using capital to subsidize costs and sell your product for less than it's worth actually seems like a huge market failure to me, and is the consequence of huge concentrations of capital in the hands of a small and shrinking group of people. The only people who can afford to compete are the already wealthy. It stifles competition.

As a consumer it should definitely concern you if you're using poorly financed or unsustainable products. Maybe it's great now but I would rather have a slightly more expensive product that is more stable than one that suddenly had to make the experience a lot worse because they burned a lot of cash and had a down round.

>It's a well-established, economically rational business model to subsidize prices in order to gain market share.

Economically rational for the business owner, maybe, but it inevitably results in eventual enshittification for the consumer. (And Uber isn't a great example of this; its share price has only recently surpassed what it was at IPO 4 years ago, and many companies don't even do that well)

> If the service you use goes out of business, it's pretty easy to switch to another.

I'm not sure how well that works if vendors differentiate themselves through proprietary models and backend integrations. As far as I know, Mistral only releases weights for some of its models, with others being proprietary.

Reply on news.ycombinator.com