Hacker News Reader

Comment by Dave3of5 | Hacker News Reader

Interesting numbers. That roughly equates to about $250 million per year plus I don't know how much training is costing them to keep the model up to date and suchlike.

The company also has about 375 employees. I've no idea how much they get paid but I used $200k as a yearly cost and that comes to $75 million.

That's about 3:1 cost of operating the services to paying employees. That seems quite high as I've never been at a company that had 1:1 costs for running servers vs employee costs but I could entirely be off base here.

Given Sam Altman's recent comments on the days of these LLM being over I think maybe Microsoft or whomever is basically saying that they can't spend that much money and they need to control costs much more heavily.

View on HN

They're interesting numbers, but the linked article's cite amounts to:

> ChatGPT could cost OpenAI up to $700,000 a day to run due to "expensive servers," an analyst told The Information.

which, pardon me, but no shit.

Before I break out my back of the envelope calculator, on how many biggest GPU instances in Azure that is, the real question is what their underlying assumptions are, and where they're getting them from. Especially since OpenAI is definitely not paying list price on those GPU instances.

The other question is how close to capacity their cluster is running, and how much free time on it can be reclaimed, either in terms of spinning down servers for diurnal patterns, or in terms of being able to do training runs in the background.

Considering that Microsoft is a huge investor in OpenAI, I'd be surprised they pay anything at all in reality.

that's ridiculous, OpenAI is paying. Granted Microsoft invested heavily into OpenAI but those are two separate financial transactions. Sure you can rationalize in your head that IN-OUT=DIFF but that's not how books are kept.

Why is that ridiculous? Cloud services gives companies "coupons" and free usage for X hours for a bunch of other companies, why wouldn't they do that for a company they invested heavily in?

Because that's not how it works. Even company cars of General Motors employees have to be purchased from General Motors.

Such "free usage" coupons are marketing activities to gain new customers, Microsoft already completed the "dating phase" with OpenAI. They surely don't pay list-price for Azure but it's surely also not free.

Moreover, as per Microsoft themselves, the 1bn USD investment into OpenAI carried the condition that Azure becomes the exclusive provider for cloud-services: https://blogs.microsoft.com/blog/2023/01/23/microsoftandopen...

It's not exclusive because it's free, it's exclusive because "we paid you 1bn USD to buy it from us"

I've personally worked on a project where Microsoft ate the cloud cost in order partner with us.

They might not give unfettered credits, it could be for specific projects. That said, I wouldn't be surprised if it was unfettered either.

Microsoft invested $1 billion OpenAI in 2019 and half of that amount was in Azure credits.

I'm not sure about the most recent $10 billion investment but I wouldn't be surprised if a significant amount of it is in Azure credits as well.

While that's not "free" (they exchanged equity for it), it's likely not an expense (or at least not an expense that they have to cover fully).

At best OpenAI has negotiated a near 0 profit margin for Microsoft when paying for the services. But even that is unlikely given how much money/resources are involved. There's no scenario where it's free at that scale.

The estimate in the article pins most of the cost on inference, not training, so diurnal patterns are unfortunately not as useful here.

> While training ChatGPT's large language models likely costs tens of millions of dollars, operational expenses, or inference costs, "far exceed training costs when deploying a model at any reasonable scale," Patel and Afzal Ahmad, another analyst at SemiAnalysis, told Forbes. "In fact, the costs to inference ChatGPT exceed the training costs on a weekly basis," they said.

Why wouldn't inference follow diurnal patterns?

Agreed. That seems backwards. Training would not follow circadian rhythms, inference would.

I should clarify: training is not latency sensitive, so you can run your workloads at off-peak hours when people are asleep. Inference means you need to run your workloads at peak when people are using your service.

(Looking back, I'm happy that I was careful in my wording in that I didn't say diurnal cycles aren't relevant, just that they aren't as useful in this case)

That said, I suppose I misread the specific suggestion about spinning down servers off-peak and was thinking more about spot pricing at peak vs trough.

200k is too small, strong sde1s at Amazon get paid that much in hcol areas. Closer to 500k.

This, there's like a endless line of companies waiting to snatch OpenAI's employees right outside the door. $200k average comp at OpenAI would be laughable.

As a side, I am a bit shocked by these numbers. Is this an American thing? I understand myself to be good software engineer with good well rounded experience of 14+ years. Yet my income, in Europe, is really above 100k.

What I am wondering, for those earning 500k, how big is your work load/stress. Would this be a 9-5 job you leave at the office when going home. Or does a job that earns so much consume your life?

Honestly, depends. Some teams at FAAMNG are really stressful and if you work on a Tier 1 service even with loads of SRE support you have to be working a fair bit. That being said, the pay is for design decisions at the higher IC level (senior or staff) and most people at that level are very smart. I’m not saying this salary is for 10x engineers or anything.

I would say 50% the work is harder and consuming and then 50% they can just afford to pay you more and lock up talent because of the wild margins on their products.

Amazon has a terrible reputation for internal infrastructure issues, with "on call" being a truly shitty experience for employees. aka burn out over a year is common

Note that there's likely to be some variation per team, but Amazon is famously bad, so ... ;)

FAANG salaries are so bloated because Bay Area housing costs are insane. Someone making 500k could put half or more of into their mortgage.

I've said it before on here, but I live very comfortably in Philly for a lot less than that.

I'd argue it's the opposite. We're coming off a decade of free money driving a second tech boom.

If interest rates stay elevated, and value investing becomes valuable again, it will be interesting to see how the tech space transforms. When start-ups have to compete with money market funds or even treasuries for investor cash, things become orders of magnitude tighter and more challenging.

I’ve been through both horror (endless 100 hour weeks) and bliss (just attending meetings and not really stressing about much of anything) in that range. It’s highly variable.

> Is this an American thing?

Yes, though Switzerland approaches it. If you want to see how much people of various levels of experience get paid at different companies and in different locations go to levels.fyi

Americans get paid much, much more than anyone else.

Your standard of living might be comparable. Your retirement is taken care of, you have a reasonable amount of vacation, you have better job security, your health care, in most European countries, has much less hassle, and your property costs are lower.

I am seriously considering a move if my husband can find an academic job over there. The retirement won't be a great lure (fewer years in the system) but we almost have enough to coast from here, so it's about the rest.

American SWE salaries can be insane, but I'm shocked at how low SWE salaries are in Europe.

I was expecting salaries to cool off a bit with the massive wave of layoffs across the industry, but from what I've seen, that hasn't happened.

Taxes in the bay area can be insane - ~40% if I remember correctly. On top of that you have crazy-expensive healthcare, and crazy expensive housing costs.

~100k€ in (western) Europe may be comparable to ~200k€ in Bay Area.

These numbers are insane to me.

I'm 20 years into programming and a senior architect and lead on an enterprise project.

I don't even make that first number.

But I value certain things way more than other things, and my current job provides it. Fully remote, leaves me completely alone to accomplish what they need done (and I get it done), unlimited vacation, great benefits, zero pointless meetings (almost an empty calendar).

I'm sure these other companies offer some of that but 500k?! That is absurd.

Someone had posted their tax filings in a different OpenAI thread. Although this only starts at 2020, this may give some insight into their employee costs https://projects.propublica.org/nonprofits/organizations/810...

Interesting, the 2020 revenue and costs are significantly lower than previous years. Actually the prior years give a much better insight into salaries there. I wonder if this is because they switched from the non-profit model to the for-profit subsidiary at that time?

Pretty sure he said the days of models getting larger were over. Not that LLMs we’re over

I'm not familiar with Sam's comments re: "days of these LLMs being over" - can you provide more context (or link)?

"“I think we’re at the end of the era where it’s gonna be these giant models, and we’ll make them better in other ways,” Altman said.

He sees size as a false measurement of model quality and compares it to the chip speed races we used to see. “I think there’s been way too much focus on parameter count, maybe parameter count will trend up for sure. But this reminds me a lot of the gigahertz race in chips in the 1990s and 2000s, where everybody was trying to point to a big number,” Altman said.

As he points out, today we have much more powerful chips running our iPhones, yet we have no idea for the most part how fast they are, only that they do the job well. “I think it’s important that what we keep the focus on is rapidly increasing capability. And if there’s some reason that parameter count should decrease over time, or we should have multiple models working together, each of which are smaller, we would do that. What we want to deliver to the world is the most capable and useful and safe models. We are not here to jerk ourselves off about parameter count,” he said."

via https://techcrunch.com/2023/04/14/sam-altman-size-of-llms-wo...

Sam Altman didn't say LLMs are over. (He's the CEO of OpenAI, so that would be a really strange thing for him to say, wouldn't it?)

What he actually said was that we've reached the point where we can't improve model quality simply by increasing its size (number of parameters). We'll need new techniques to continue to improve.

A typical OpenAI engineer gets 200-300k cash and 300-500k equity, at least from the levels.fyi data.

Reply on news.ycombinator.com