Hacker News Reader

Comment by tmountain | Hacker News Reader

It’s an interesting situation because on one level, the technology is at least somewhat misrepresented (best case results rising to the top of everyone’s feeds), but on the other side, we haven’t even begun to see what integrated Chat GPT will bring. If I could replace Siri with Chat GPT, I would do so immediately, as it’s objectively better. The author makes some points about the eye watering costs; however, if this is something that people truly want—and I think we do want it—the market will find a way to deliver it at scale in a cost effective way.

View on HN

I agree, it is a bit of an odd moment. On the one hand, you have the normal amount of hype and bullshit. The people calling out that hype are right of course, as they have been right about 1000 other vaporware demos and outright marketing lies. The groundbreaking product does not yet exist and may never exist.

But on the other hand, there is something real and incredible here. It has massively pushed forward the frontier of what a computer can do. If you compare the output of these models to sci-fi like Star Trek, the fantasy was too conservative. It calls into question what it means to be human, what makes us special, and what is art. It’s the most interesting moment in art since postmodernism.

So many people are rightly looking at the trend and trajectory. Incredible things have already been created. Humans have been beat at their own games. Humans performance has already been matched on a wide range of tasks. If you factor in 1 more year of progress, 5 more, 50 more, you might reach some terrifying conclusions. It’s only human.

> It calls into question what it means to be human, what makes us special, and what is art.

How?

> the market will find a way to deliver it at scale in a cost effective way

Who is this market, who is so wise in the ways of science?

Says the person typing the message on a personal supercomputer that has enough battery power to last all day and fit in their pocket at the same time....

(assuming that is, that this message came from a portable device)

Portable devices haven’t achieved cost effectiveness without dangerous conditions, slavery and child labor on the other end as externalized costs

All costs are externalized by the premature heat death of the universe.

And I mean really, if this is the bar you want to set then no product is going to meet those goals, so no huge point in pointing out electronics, versus say just about anything you eat on a given day, or the clothes you wear.

uh - no product is going to meet the goals of avoiding slave and child labor and hazardous work conditions? not sure why you're going to bat for that besides to defend that it's hugely profitable. i think many products outside portable devices do meet that goal but it's not in service of market competition and isn't a scalable solution without a bigger systemic change, so i do agree in a sense with saying that at scale no product meets that goal sure.

Honestly, if the price of a smartphone was even $10,000, I would be happy to pay it. The value I get out of it is incredible. In my case (living in a mid-sized city), if I had to choose between my phone and my car, I'd definitely sell the car first.

well solutions for these problems are not going to come from consumers independently deciding to pay more to "green wash" their purchases

You might enjoy SiriGPT: https://gpt3demo.com/apps/chatgpt-sirigpt-apple-ios

(It's voice-activated ChatGPT on your iOS device using shortcuts.)

> the market will find a way to deliver it at scale in a cost effective way

This really isn't a given. Look at the problems encountered trying to scale Bitcoin - slow transaction speed and high energy use. They're still unsolved, because some properties of a given technology are inherent to that technology.

There's no guarantee something is scalable, especially something that might (as the article suggests) have exponentially increasing costs in its default configuration.

> Look at the problems encountered trying to scale Bitcoin - slow transaction speed and high energy use.

They're not unsolved, they're actually trivially solved by using an exchange. That just comes with its own risks.

How is the high energy cost of mining Bitcoin solved by exchanges? Also if I want to use crypto for anything except speculation within an exchange, the transaction speed is still an issue, even with apps like Coinbase. (Or at least it was last time I used it).

I know Lightning Network helps but that's like a layer on top of Bitcoin because the Bitcoin tech itself doesn't scale well

> the market will find a way to deliver it at scale in a cost effective way

I wonder if it would be possible to build the model in hardware as an ASIC. That would probably bring down cost a lot but I'm not sure it makes sense if they expect to release new improved models regularly. The hardware might be obsolete by the time it reaches production.

I think that if we see hardware specific to a particular model it would be in automotive. And the focus would be on fitting inside a power envelope, not cost in particular. So it still could be that cost of asic + operation will be greater than gpu/tpu + operation, but it will fit in whatever limit (1500 watts? Idk) is designed for driving systems more easily.

Recently wrote about the economics of both training and inferencing: https://sunyan.substack.com/p/the-economics-of-large-languag...

Submit it as a top level post. It’s a good article.

Maybe eventually, but there are plenty of examples of things taking quite a long time before they are economically viable.

Do we have an idea of what the costs to run ChatGPT might be? Are we costing OpenAI $5 everytime we ask it to generate python in the style of Donald Trump?

Probably training it on specific data sets and contexts is the expensive part.

Running a gigantic model across multiple GPUs isn't cheap either

I’ve asked ChatGPT about hardware. Here’s the response:

> As for your specific computer, with 64 GB of RAM and a high-performance GPU like the GeForce 1080 Ti, it should have sufficient resources to run a language model like me for many common tasks.

Based on the models open sourced by OpenAI, they are using PyTorch and CUDA. This means their stack requires nVidia GPUs. I think the main reason for their high costs is a single sentence in the EULA of GeForce drivers: https://www.datacenterdynamics.com/en/news/nvidia-updates-ge...

It’s technically possible to port their GPGPU code from CUDA somewhere else. Here’s a vendor-agnostic DirectCompute re-implementation of their Whisper model: https://github.com/Const-me/Whisper

On servers, DirectCompute is not great ‘coz Windows server licenses are expensive. Still, I did that port alone, and spent couple weeks doing that.

OpenAI probably has resources to port their inference to vendor-agnostic Vulkan Compute, running on Linux servers equipped with reasonably-priced AMD or Intel GPUs. For instance, Intel A770 16GB only costs $350, but delivers similar performance to nVidia A30 which costs $16000. Intel consumes more electricity but not by much, 225W versus 165W. That’s like 40x difference in cost efficiency of that chat.

I've read estimates of $0.02 per query

Guesstimated around $0.01 per message

https://mobile.twitter.com/tomgoldsteincs/status/16001969953...

So way cheaper than an SMS not so long ago

SMS are free (they use up left over space in the signalling protocol a phone needs to use to talk to towers/inform the network of where it can be found - a process you're already charged for). Doesn't seem like phone networks collecting money for no effort (which is why it trends towards free) is a fair comparison to running an expensive supercomputer with very real hardware, energy and real estate costs.

We can probably make a decent guess by looking at the price they charge for GPT-3, where asking the most expensive model for its literal "two cents" response gives you a thousand tokens.

I couldn't resist and tried it:

Prompt: Write a small python program in the style of Donald Trump.

chatGPT: I'm sorry, but it wouldn't be appropriate to write a program in the style of a political figure, especially if the language and tone used may be considered offensive or disrespectful. Additionally, OpenAI's policy prohibits the creation of content that is harmful, abusive, or hateful. Is there anything else I can assist you with?

Reply on news.ycombinator.com