Using GPT-4 to generate 100 words consumes up to 3 bottles of water | Hacker News Reader

Using GPT-4 to generate 100 words consumes up to 3 bottles of water

tomshardware.com

13 pointsgeox13 comments

Consumes? As in, the water disappears?

Also, it's pretty incredible. A very comparable job can be done with a local model on my MacBook Air in few minutes. How many bottles of water is that? That's so gigantic, immeasurable difference in efficiency I can't help but think the article is totally wrong.

BTW I used ChatGPT today. If I didn't, I would have to spend a week doing menial work. Instead, the job was done in few hours. How many bottles of water it saved from being pissed away by me? None, I still piss the same amount of water, but now I have a better ratio of job done vs water pissed.

What a weird way to measure something...

Err… no. They seem to be using the power consumption of an entire data centre as if the whole building was used for just your email. That’s not how that works!

GPT-4 needs roughly 8x GPUs and can generate 100 words in about a second or two. Let’s say it’s a single DGX server running for 2 seconds at 6.5kW. This is just 13 kJ, which can be cooled with about 5mL of water maximum… not 3 bottles!

In reality it’s much more efficient than that because during inference multiple requests are processed together, typically something like 8 to 32 at once depending on the model and the hardware. This means that a single email needs something like a tiny drop of water… which isn’t destroyed, merely evaporated.

This is histerical reporting by a scientifically illiterate arts major worried about being replaced by an AI next year.

Source for gpt4 requiring 8 gpus?

Rumours are that the original base model was a 16-way mixture of experts over 1 TB in size. With quantisation that would fit into 8x 80GB cards = 640 GB total. That is similar in size to competitive models. There’s also a practical benefit of being able to use 100% of a common platform such as a DGX server. Splitting across multiple servers is much more complex and less efficient.

Do you need to run all experts at once?

No, but different ones are invoked for each word so typically they’d all be preloaded into VRAM. Also, because of batching, different users would use different experts concurrently.

There’s no detailed information about how GPT4 is hosted, but we can guess from competing models and the leaked info.

This is one of those things that only makes sense in comparison to other objects.

How much water a requires a bag of almonds grown in California?

A gallon of water per almond.

https://www.motherjones.com/environment/2015/01/almonds-nuts...

I know, I am engaging in “whataboutism” but I also feel like articles like this are harmful. It is a red herring.

Often times, I find claims of "whataboutism" to be unfounded. In this case, the editor chose to present a specific quantity in the article's header. Now, this quantity may be accurate (I don't know why someone would lie about this), but I would assume that most readers would not know how to put it into context. And I would also assume that the editor likely was aware that most readers would not know how to put this into context and that they would think that the number was really big. And your comment provided a useful reference point - a lot of things probably use gallons of water, and maybe the obvious narrative you would get from the article would not be entirely accurate.

Evaporative cooling (and the water required) is cheaper than the electricity for traditional cooling. If the economics changed or there really wasn't enough water, they could change how the data center is cooled. This is mostly a fake problem.

There's a data center nearby and every day you can see the pile of empty bottles grow. It's tragic.

What does the water get transformed into? Coal? Dirt? Gold? Carbon monoxide? Neutrinos?

Seems a bit hysterical.

I, a human, only need a few sips for 100 words. Humans 1, AI 0.

Reply on news.ycombinator.com