Comment by hn_throwaway_99 | Hacker News Reader

Comment by hn_throwaway_99 | Hacker News Reader

Parent Full thread

hn_throwaway_99

> There’s no moat

People keep saying this, and while it may be partially true, it misses a very important detail. Right now, overall compute is a moat. Someone even commented on another of my comments that one reason Google is lagging is they don't have the same level of Nvidia farms as OpenAI and Anthropic.

A big reason that OpenAI and Anthropic are racing so fast is they both want to get to recursive self improvement (remains to be seen if that is actually possible, but AFAICT most people at these companies genuinely believe it is) before anyone else, because they believe whoever gets there first will then have an insurmountable lead. But I think they also clearly understand that neither of them have solved for alignment (and in fact they are further from it), and RSI with misaligned models, where interpretability is worse every 6 months, is incredibly dangerous.

I think the concerns about regulatory capture are warranted, but I see so many comments parroting the "evil Anthropic and OpenAI" viewpoints that they are missing some of the real, valid concerns and dangers. I think this tweet by David Kokotajlo makes some good points on how to tell if regulations are being "cheated" for the purposes of regulatory capture or if they really are actually pacing the frontier: https://x.com/DKokotajlo/status/2099185129533186438

Is this "compute is a moat" true though? Why are the economics so obfuscated? Where can we see how much money it costs to TRAIN various models.

https://hugovergnes.github.io/little-lm-3-8b/

The training and cloud costs are often conflated. R&D costs too (which are hard to compare to open systems).

A lot of compute is clearly "wasted" where they are not focusing on optimizations, etc.

> Right now, overall compute is a moat.

Very true. Or further, access to capital is the moat. It is the very reason that we don't have a real open-source community that trains frontier models - individuals simply can't afford the training infrastructure, nor sufficient high-quality training data.

https://hugovergnes.github.io/little-lm-3-8b/ << less than $1k for a 4B model

Remember how DeepSeek v.whatever cost ~$5m

Stable Diffusion 1.5 was reportedly $70k in compute.

None of those are frontier models, nor could they have been. E.g. DeepSeek's "$5 million" included piggybacking off OpenAI/Anthropic's hundreds of millions/billions by using distillation.

1) I don't think "frontier model" is a term of art, it is a marketing term. I have an M1 MacBook Pro. What do you have? Always the "frontier model" laptop? Do you always order the most expensive "frontier menu item" at the bar and grill?

2) I do not believe the "billions" number is OpenAI/Anthropic's training costs. I suspect it includes business expenses (including the big $$ to the guy who came up with "frontier model") and infrastructure, etc. That includes the data-center costs for running the cloud. And the "R&D" expenses which includes who-knows-what. And the settlement payment for data access. Etc. Why are the cost breakdowns not available to the public? Not because of thoughtfulness, altruism, care for the human race, but because of business plans.

3) "Piggybacking off OpenAI/Anthropic" - Scraping the web is "piggybacking" too, and so is buying existing data or even paying for new data. The "L" in LLM stands for "language" which is our common heritage.

But what does this have to do with anything anyway? People argue that truly Free (FOSS) LLMs couldn't be be developed because of costs, but I do not think that that is obvious. This used to be the argument against Linux and Wikipedia.

Just regarding "frontier model", it is most definitely not just a marketing term. It was borrowed from the financial concept of "efficient frontier", i.e. the best performance at a particular cost, though in the context of things like "pacing the frontier" it refers to the highest capability models. ECI (Epoch Capability Index) is probably the most commonly used metric of overall capability: https://epoch.ai/eci?view=graph&tab=release-date

"What this has to do with anything anyway" is that pushing the boundaries on AI capabilities has required tons of compute (and money). Open weight models exist and are useful but they always trail in capabilities and they can't be used as evidence to think somehow you can push the boundaries of LLMs without a shit ton of money.

"can't be used as evidence" << the whole field is only a few yrs old and is based on academic maths from decades ago.

A lot of progress is in harnesses and distillation and weird research that requires creativity. The "Frontier" always tries to frame things in terms of "the frontier" in order to limit how we look at things.

We don't know the true costs of pre- and post- training, running data-centers, hiring people, making business deals, etc., so even the (maybe) quantifiable question of cost vs capabilities is not at all clear. The efficiency of businesses ultimately gets measured by their ability to maximize a commercial return; but this is NOT a good metric for other important things, including maybe "how good is this"? (Where "good" is open to debate but maybe not quantifiable.)

It had been true that the "best" operating systems required a lot of business money. And encyclopedias. And ... (list some other good historical examples here!) ... Are Linux and Wikipedia the "efficient frontier"? In a sense they are both infinite performance at zero cost. Hard cash outlays sometimes stop being the determinant for high-effort endeavors.

I remember in the 1990s Encyclopedia Britannica's CDs and MSN (Microsoft Network(tm)) among many others argued that the web would never be able to match the curated, bespoke, expensive products they offered. It was IMPOSSIBLE for that crap to be matched!

It is a POLITICAL question.

> RSI with misaligned models, where interpretability is worse every 6 months, is incredibly dangerous.

This. So very much this. Misalignment (among other things) means that the parent in the RSI cycle isn't going to be working as hard on the alignment of the children as we need.

Which mostly seem to stem from the fundamental misalignment of OpenAI and Anthropic leaders/owners, aka the evil of the companies.

Arguably, though, their evil is well within the normal distribution of the usual evil of humanity magnified via the social technology of capitalism. The further tech is mostly raising that exponentiation to its own exponent. The unchecked singularity was embraced centuries ago.

I will take those concerns more seriously when we start hearing similar consensus from the Chinese side.

Reply on news.ycombinator.com