Hacker News Reader

Comment by grues-dinner | Hacker News Reader

> For all the desperate founders rushing to train their models to convince their investors for their next $100 million round.

Has anyone actually trained a model actually worth all this money? Even OpenAI is s struggling to staunch the outflow of cash. Even if you can get a profitable model (for what?) how many billion dollar models does the world support? And everyone is throwing money into the pit and just hoping that there's no technical advance that obsoletes everything from under them, or commiditisation leading to a "good enough" competitor that does it cheaper.

I mean, I get that everyone and/or they investors has got the FOMO for not being the guys holding the AGI demigod at the end of the day. But from a distance it mostly looks like a huge speculative cash bonfire.

View on HN

> For all the desperate founders rushing to train their models to convince their investors for their next $100 million round.

I would say Meta has (though not a startup) justified the expenditure.

By freely releasing llama they undercut every a huge swath of competition who can get funded during the hype. Then when the hype dies they can pick up what the real size of the market is, with much better margins than if there were a competitive market. Watch as one day they stop releasing free versions and start rent seeking on N+1

Right, but that is all predicated that, when they get to the end, having spent tons of nuclear fuel, container shiploads of GPUs and whole national GDPs on the project, there will be some juice worth all that squeeze.

And even if AI as we know it today is still relevant and useful in that future, and the marginal value per training-dollar stays (becomes?) positive, will they be able to defend that position against lesser, cheaper, but more agile AIs? What will the position even be that Llama2030 or whatever will be worth that much?

Like, I know that The Market says the expected payoff is there, but what is it?

As the article suggests, the presence of LLAMA is decreasing demand for GPUs. Which are critical to Metas ad recommendation services.

Ironically, by supporting the LLM community with free compute-intense models, they’re decreasing demand (and price) for the compute.

I suspect they’ll never directly monetize LLAMA as a public service.

With all these billions upon billions in AI hardware screaming along, are ads actually that much better targeted than they used to be?

I imagine admongers like Meta and Google have data that shows they are right to think they have a winning ticket in their AI behemoths, but if my YouTube could present any less relevant ads to me, I'd be actually impressed. They're intrusive, but actually they're so irrelevant that I can't even be bothered to block them, because I'm not going to start online gambling or order takeaways.

A better question, with a growing push for privacy, how can they keep ads from regressing?

There’s a lot more that goes into the ad space than just picking which ad to show you, and it’ obviously depends on who wants to reach you. For example, probabilistic attribution is an important component on confirming that you actually got the ad and took the action across multiple systems.

Also, since you mentioned it, TV ads tend to be less targeted because they’re not direct-action ads. Direct action ads exist in a medium where you can interact with the ad immediately. Those ads are targeted to you more, because they’re about getting you to click immediately.

TV ads are more about brand recognition or awareness. It’s about understanding the demographic who watches the show, and showing general ads to that group. Throw a little tracking in there for good measure, but it’s generally about reaching a large group of people with a common message.

You ask a great question, and I wonder how the push for more privacy will pan out (pardon the gold mining analogy). I am almost done with the very good new book The Tech Coup by Marietje Schaake, and I have also read Privacy is Power and Surveilance Capitalism. I think more of the public is waking up to the benefits of privacy.

All that said, I am an enthusiastic paying customer of YouTube Prime and Music, Colab (I love Colab), and sometimes GCP. For many years I have happily have told Google my music and YouTube preferences for content. I like to ask myself what I am getting for giving up privacy in a hopefully targeted and controlled way.

> Ironically, by supporting the LLM community with free compute-intense models, they’re decreasing demand (and price) for the compute.

For other people that that sentence didn't make sense for at first glance: "by supporting the LLM community with free compute-intense models [to run on their own hardware] they’re decreasing demand (and price) for the compute [server supply]."

Sorry, I should have been more clear.

They’re decreasing demand for expensive GPUs that would be required to train a model. Fine-tuning and inference are less compute intense, so overall demand for top-end GPU performance is decreased even if inference compute demand is increased.

Basically, why train an LLM from scratch, and spend millions on GPUs, when you can fine tune LLAMA and spend hundreds instead.

Thank you for the extra clarification, I hadn’t even thought of inference vs training!

How fungible is that compute though? Having even a single H100 is different than having a bunch of 4090's, nevermind a properly networked supercomputer of H100s.

That’s the point. You can run inference on a 4090 but training is better on a H100. If you use llama, you don’t need to train on an H100, so you can free that supply up for meta.

I haven't been following llama closely but I thought the latest model was too big for inference on 4090's, and that you can't fine tune on 4090's either, but furthermore, the other question is if the market is there for running inference on 4090s.

Well, (1) there are a ton of GPUs out there of various specs, and you can also use an inference provider who can use a H100 or similar to serve multiple inference requests at once. (2) there are a ton of LLAMA sizes, from 1b, 2b, 8b, 70b, and 400b. The smaller ones can even run on phone GPUs.

> having spent tons of nuclear fuel

It will be primarily gas, maybe some coal. The nuclear thing is largely a fantasy; the lead time on a brand new nuclear plant is realistically a decade, and it is implausible that the bubble will survive that long.

> there will be some juice worth all that squeeze.

Without the squeeze there'd be a risk for some AI company getting enough cash to buy out Facebook just for the user data. If you want to keep status quo it's good to undercut someone in the cradle that could eventually take over your business.

So it might cost Meta pretty penny but it's a mitigation for existential risk.

If you climbed up to the top of wealth and influence ladder you should spend all you can to kick off the ladder. It's gonna be always worth it. Unless you still fall because it wasn't enough.

Given their rising stock price trend, due to their moves in AI. Definitely worth it for them

Given meta hasn’t been able to properly monetize WhatsApp I seriously doubt they can monetize this.

Who says they haven't?

> I get that everyone and/or they investors has got the FOMO for not being the guys holding the AGI demigod at the end of the day

Don't underestimate the power of the ego...

Look at their bonfire, we need one like that but bigger and hotter

I spit out my tea when I read your last sentence. You should consider standup comedy.

It's a bonfire, turn the lights out

Isn’t OpenAI profitable if they stop training right at this moment? Just because they’re immediately reinvesting all that cash doesn’t mean they’re not profitable.

And if they stop training right now their "moat" (which I think is only o1 as of today) would last a good 3 to 6 months lol, and then to the Wendy's it is.

That is similarly true for all other AI companies. It’s why they don’t do that. But everyone is still happy to give them more money because their offering is good as it is.

>and then to the Wendy's it is

I didn't really catch that pop culture reference. What does that mean?

My guess: The competition catches up, you lose all paying clients, and you get to apply for jobs at Wendy's...?

Or does it mean it's a trivial decision, as trivial as deciding what fast food joint to choose?

Guess we'll never know :(

This guy claims they are losing billions of dollars on free ChatGPT users:

https://nitter.poast.org/edzitron/status/1841529117533208936

Ed Zitron's analysis hinges on a lot of assumptions. Much of it comes down to the question of how much it actually costs to run a single inference of ChatGPT. That $20/month pro subscription could be a loss-leader or it could be making money, depending on the numbers you want to use. If you play with the numbers, and compare it to, say, $2/hr for an H100 currently on the front page, $20/$2/hr gets you 10 hours of GPU time before it costs more in hardware than your subscription, and then factoring in overhead on top, it's just not clear.

You’d need to know how much they are using for that. I only use the API and the $20 I bought a year ago aren’t gone yet.

Not everyone is doing LLM training. I know plenty of startups selling AI products for various image tasks (agriculture, satellite, medical...)

Yes, a lot of the money to be made is in the middleware and application sides of development. I find even small models like Llama 3.2 2B to be extremely useful and fine tuning and integration with existing businesses can have a large potential payoff for smaller investments.

Lots of companies have. Most recently Character AI trained an internal model and did raise $100M early last year. They didn't release any benchmarks since the founding team and Noam taken to Google

Pretty sure anthropic has

Reply on news.ycombinator.com