Hacker News Reader

Comment by 999900000999 | Hacker News Reader

This was an interesting article, but he completely misses the only real threat to either Open AI or Anthro.

Open source models like deepseek and llama 3 are rapidly catching up, if I can get 90% of the functionality for significantly less ( or free if I want to use my own GPU), what value does open AI really have .

I'm a paid subscriber of open AI, but it's really just a matter of convenience. The app is really good, and I find it's really great for double checking some of my math. However I don't know how they're going to ever truly become a corporate necessity at the prices they're going to need to bill at to become profitable.

Then again, open AI is obviously ran by some of the smartest people on planet Earth, with other extremely smart people giving them tons of money, so I can be completely wrong here

View on HN

> Open source models like deepseek and llama 3 are rapidly catching up

Catching up for how long though? Large models are very expensive to train, and the only reason any of them are open is because ostensibly for-profit companies are absorbing the enormous costs on behalf of the open source scene. What's the plan B when those unprofitable companies (or divisions within companies) pull the ladder up behind them in pursuit of profits?

> Catching up for how long though? Large models are expensive to train and the money has to come from somewhere, and for now that somewhere is the charity of ostensibly for-profit companies which release their models for free and just eat the losses. That doesn't seem very sustainable to me.

Catching up enough to execute a business successfully on the open source (and free as in free beer) alternatives to OpenAI. Once you have a single model that works, you can bootstrap your own internal ML infra for just that one use case and grow from there.

Martin Casado from a16z stated an opinion that 80% of US startups are likely using less expensive open models, usually from China. Chamath P. on the All In Podcast said that his company is using Chinese models, but hosted in the US and he cited his company as a huge inference user.

Important to distinguish that they're likely using them for internal workflows (agents, etc) where the scope is well defined and they can tune their prompts and evals to accommodate a lower-performance model.

Nobody is advocating switching their coding agents to open source (yet), but that's not the bulk of the tokens in companies that have automated workflows integrated into their business.

LLMs can be trained using a distributed approach [1]. So what’s stopping the OSS community from setting up a pool in the likes of seti@home and boinc?

[1] I asked Gemini and apparently it’s quite common

And then imagine Chinese do distributed training at home. What outcome can they achieve!

I think what you're missing here is that China would be (is?) happy to fund the development, because it's in their national interest and necessary in order for their companies to stay competitive, so long as there are trade restrictions on chips. Another framing for this is that China and certain other entities (e.g. content distribution channels like Meta, Youtube) have a strong incentive to 'commoditize their [AI] complement' (https://gwern.net/complement).

Listen to one of the Zuck interviews, he explains this everywhere.

To paraphrase, Meta is not releasing open models out of the goodness of their heart, they're doing it as part of their business strategy.

- They already have all the eyeballs.

- They want to be able to run the best model they can for those eyeballs.

- It doesn't make a difference to them whether others have a similar model or not. They don't care.

- Ergo, they release their weights, hoping someone can help them improve it. Which has happened: see quantization.

I don’t think that’s the point he is making. The argument to me is looking at the numbers and grounding them in real life. It takes time to build data centers, it takes people to run them. The article makes the argument that the timelines are not feasible.

This is such a techno-centric view, you're not even remotely aware of your own biases.

> but he completely misses the only real threat to either Open AI or Anthro.

Hard disagree. Economics matter, in fact more than tech.

Tech only gets to shine if the economics work out.

> Then again, open AI is obviously ran by some of the smartest people on planet Earth, with other extremely smart people giving them tons of money, so I can be completely wrong here

Nope.

Money != Intelligence

Open AI, SF and the West-Coast VC scene is run by very opionated, incentiviced people.

Yes, money can make things move, but all the money of the world don't matter if your unit econonmics don't work out.

And the startup graveyard is full of examples of this kind.

> free if I want to use my own GPU

The hardware required to run something like deepseek / kimi / glm locally at any speed fast enough for coding is probably around $50,000. You need hundreds of gigabytes of fast VRAM to run models that can come anywhere close to openai or anthropic.

$50k would be the cost to run it un-quantized, 10k could get you for example 4 5090 system, that would run the 671b q4 model which is 90% as good, which was the OPs target

which 671b quants can fit into 96GB VRAM? Everything I’m aware of needs hundreds at least (e.g. https://apxml.com/models/deepseek-r1-671b).

5090 is 32 GB so it's 128GB, not 96.

128 is still not 300. Something like 4x 6000 blackwell is the minimum to run any model that is going to feel anything like claude locally.

To my deep disappointment the economics are simply not there at the moment. Openrouter using only providers with zero data retention policies is probably the best option right now if you care about openness, privacy and vendor lock-in.

For local use and experimentation you don't need to match a top of the line model. In fact something that you train or rather fine-tune locally might be better for certain use cases.

If I was working with sensitive data I sure would only use on prem models.

Meanwhile, the money asks, short time frames, and unclear demands for same are huge business problems. Self service (llama on my own gpus) I suppose is just another way to ask where the demand is coming from to argue for billions in new money. Something smells ...

> I'm a paid subscriber of open AI, but it's really just a matter of convenience. The app is really good, and I find it's really great for double checking some of my math.

That right there is why they are valuable. Most people are absolutely incompetent when it comes to IT. That's why no one you meet in the real world uses ad blockers. OpenAI secured their position in the mind share of the masses. All they had to do to become the next google was find a way to force ads down the throats of their users. Instead they opted for the inflated bubble and scam investors strategy. Rookie mistake.

The mind share openAI has is next to none.

The reality is they're a paid service, and even if they 10x their prices they're still in the red.

Consumers do actually care about price. They will easily, and quickly, move to a cheaper service. There's no lock in here.

There is talk of 800 million weekly users or whatever. But real question to me is how much actual disposable income they have or willingness to spend it on expensive AI subscription.

Not true, for the non tech crowd ChatGPT is the AI. There are a few people using Grok or Gemini, fewer outside the coding crowd would know anthropic

This is just not true. They don't even know what an OpenAI is, they just know what chat is. It's a chat window.

You make another chat window and you're golden.

Ok make one and show us your market share. I’m talking about mind share and ubiquitous. Who’s going to your talk.ai no one.

I said everyone knows ChatGPT you responded about OpenAI

It's a chat window, it's already ubiquitous.

The downside to making a zero-effort product is that consumers can easily just switch to something else with no thought. ChatGPT has literally nothing that makes it unique.

Almost a 3rd of users use ad blockers

https://backlinko.com/ad-blockers-users

And just because you have users doesn’t mean it’s easy to create a profitable ad business - ask Yahoo. Besides we still don’t know how much inference costs. But there is a real marginal costs that wouldn’t be covered by ads. They definitely couldn’t make enough on ads to cover their training costs and other costs.

And adding ads into the responses is _child's play_ find the ad with the most semantic similarity to the content in the context. Insert at the end of the response or every N responses with a convincing message that based on our discussion you might be interested in xyz.

For more subtle and slimier way of doing things, boost the relevance of brands and keywords, and when they are semantically similar to the most likely token, insert them into the response. Companies pay per impression.

When a guardrail blocks a response, play some political ad for a law and order candidate before delivering the rest of the message. I'm completely shocked nobody has offered free gpt use via an api supported by ad revenue yet.

For actual work and not toying around the 10% gap is absolutely worth the cost.

You may be correct, but for my hobbyist projects I tend to use the cheaper models to get started, and then I'll switch to a more expensive one if the cheaper model gets stuck.

Unless the actual race is to create an AI Employee that operates and can deliver work without constant supervision. At that level, of course it would be cheaper to pay $2,000 a month straight to Open AI vs hiring a junior SWE

> For actual work and not toying around the 10% gap is absolutely worth the cost.

I agree with this... for now. But the hosted commercial models aren't widening the gap as far as I can tell, if anything it appears to be narrowing.

And if the relative delta doesn't increase somehow I don't see any way in which the "AI race" doesn't end in a situation where locally run LLMs on relatively cheap hardware end up being good enough for virtually everyone.

Which in many ways is the best possible outcome, except for the likely severe economic effects when the bubble bursts on the commercial AI side.

Sure. I'm talking about now.

I don't know what will happen tomorrow, let alone 1 year or more down the line.

If it ever becomes economical to run and maintain bare metal GPU compute to run LLMs, then that's what will need to be done..

Yes. But there is not only OpenAI. There is Gemini, Grok, whatever. If it doesn't become a "the winner takes it all" but a commodity like web hosting, then the payoff breaks down.

I feel like options for local inference are getting better. AMD has their Strix Halo. Intel's next CPU generation Arrow Lake will have better inference abilities as well.

How clearly are you able to define “actual work” and “toying around?”

Or is this the case of every HN discussion where what you do is “actual work” and what other people do is “toying around?”

Are you making [actual] money? That's literally it.

I don’t know why you’re downvoted. It’s a good and fair answer. I disagree with it to some extent but at least it makes sense.

No worries. I stopped caring about points and validation years ago lol just being honest is enough for me.

This highly depends on the price difference and value you get form those 10%.

Don’t underestimate the cost of getting locked into a tool that is 100% guaranteed to rugpull you on both cost and privacy.

You know that this is literally not a concern right? It is part of business life to navigate such a situation.

When I was a SWE, I've done migrations between bare metal to AWS to GCP and then both, and then all plus Azure..

It is the cost of doing business. You pick what works best at a price that is optimized for your business needs now. You have a war chest so when they start becoming assholes, you have leverage to fight back or pivot.

I'd rather spend $200-400/mo to unblock myself NOW than do something dumb with 5 or even 100 tokens per second output that isn't actually that good as what the current providers offer. I'm going through millions of tokens a day.. I couldn't do that with local "RIGHT NOW" (<--- important)

I think you're wrong, in the same way that folks on HN were wrong about Dropbox–HN: why would I pay for something that provides so little value, it's just slightly more convenient file storage?

Just because open source models are almost as good, doesn't mean you can underestimate the convenience factor.

Both can be true: we're in an AI bubble, and the large incumbents will capture most of the value/be difficult to unseat.

> folks on HN were wrong about Dropbox–HN: why would I pay for something that provides so little value, it's just slightly more convenient file storage?

https://news.ycombinator.com/item?id=42392302

(Before discussion of your comment devolves into nonsense about this.)

On the other hand, no one has figured out how to make money providing AI yet, and everyone's operating at a loss. At some point they're going to need to monetize, and the cost/convenience compared to alternatives may not be worth it for a lot of people.

At one point you could get a Netflix subscription and it was convenient enough that people were pirating less. Now there's so many subscription services, we're basically back to cable packages, paying ever increasing amounts and potentially still seeing ads. I know I'm pirating a lot more again.

Uber vs cabs, Airbnb vs hotels - We've seen it time and time again, once the VC cashflow/infinite stonk growth dries up and they need to figure out how to monetize, the service becomes worse and people start looking for alternatives again.

Yeah, but not just that. I don't expect my mum to go find some high end consumer GPU and install it on a home server in order to run her own local LLM. I expect that people will be throwing chat interfaces running remixed versions of open weight models out on the internet so fast that it's impossible for anyone to monitise it in a reasonable way.

I also wonder whether, similar to bitcoin mining, these things end up on specialist ASICS and before we know it a medium tier mobile phone is running your own local models.

A top end iphone is capable of running smaller local LLMs today.

A few days late but the cheapest, base model iPhone, the 16e runs Apple Intelligence too

(Unfortunately... only Apple Intelligence right now...)

I don't have a newer iphone to run it on, so I don’t have a specific one to recommend but searching for local LLM in the App Store gives plenty of options so you’re not limited to Apple Intelligence models.

I don't get this comparison. The non-Dropbox version was magnitudes less convenient to 99.99% of the population. A non-OpenAI chat interface is, at best, a fracfion less convenient.

A good number of people used to pay for email. Now a tiny fraction does. It all hangs on wbether OpenAI can figure out how to get ad revenue without people moving to a free competitor without them - and there will be plenty of those.

Does it have to be ads? :/

Of course, as that's where the money and power is, the only things SamA is in it for.

There will be equivalent models that are free. Likely, there will even be free ones without ads.

Free+ads can beat free without ads on pure incumbency, marketing and convenience. Most people don't use an adblocker, even though it's trivially easy to install.

Paid, however, can't beat free+ads. Too much friction.

Well seeing how Dropbox is doing now, Steve Jobs was right - it isn’t a product, it’s a feature. For the same price of 2TB of storage on Dropbox you can get the same amount on Google or OneDrive with a full office suite.

People love to quote Dropbox ignoring all of the YC companies that are zombies or outright failed. Just looking at the ones that have gone public.

https://medium.com/@Arakunrin/the-post-ipo-performance-of-y-...

Public...? Oh, you mean the ones meant to be left holding the bag.

When there's real money to be made investing in YC is off limits to the public: https://jaredheyman.medium.com/on-the-176-annual-return-of-a...

I am not saying the companies in aggregate don’t lead to successful outcomes for VCs. I am saying claiming Dropbox is a shining example of a “successful” company that HN was wrong about long term doesn’t jibe with reality.

They also didn’t have the massive fixed cost outlays nor did they have negative unit economics that OpenAI has.

> Open source models like deepseek and llama 3 are rapidly catching up, if I can get 90% of the functionality for significantly less ( or free if I want to use my own GPU), what value does open AI really have

They pay for the hardware and electricity /s.

You have several providers who host both deep-seek and llama 3. They pay for the hardware and electricity, you pay for usage but it's significantly cheaper than using OpenAIs models.

> you pay for usage

Plenty of people don't. That's an enduring advantage of using GPT over anything locally hosted.

Where are these providers and do they offer batch processing? If they don't how does there cost compare to Gemini and OpenAI batch processing? For the hobby project I'm working on batch processing is a great fit. The only cost comparison tool I've been able to find is openrouter and it doesn't support batch processing for cost savings.

Aws?

There is no threat, both are private public partnerships that greatly benefit the military/empire. Read Pynchon.

Reply on news.ycombinator.com