Comment by jasode | Hacker News Reader

Comment by jasode | Hacker News Reader

Parent Full thread

>Which is more "democratized": a large language model which can be downloaded, and accessed as a library, [...] or an API, where every invocation is a call that flows through Cohere's servers?

I do understand the point you're trying to make: local autonomy is superior to cloud access mediated by a private commercial entity.

However at this time, we may have a counterintuitive situation where API access is more "democratic" than downloading a huge model.

Based on various reports[1], the GPT-3 model was trained on ~45 terabytes of text corpus (Wikipedia + Web Common Crawl + book texts, etc) and the final runtime model (175 billion parameters) requires ~350 gigabytes of RAM. In that case, the model size is ~1% of the training set.

So "democractize" depends on how ambitious the user is. If you want to use a very large 350GB RAM model, the cloud model with API will be more accessible by the masses than running on local hardware. Last time I looked, an Intel Xeon motherboard has max ram of 128GB so scaling up to 350GB RAM is not going to be cheap or trivial to build.

Let's further extrapolate to a future hypothetical GPT-4 using ~10x multiplier: train on 450 terabytes of text with a model requiring 3.5 terabytes of RAM. How do we make that future huge model accessible to the masses? Probably via a cloud API. Unfortunately, there's an unavoidable hardware capital expense barrier there.

[1] https://www.google.com/search?q=gpt-3+350gb+ram

> Last time I looked, an Intel Xeon motherboard has max ram of 128GB so scaling up to 350GB RAM is not going to be cheap or trivial to build.

Even a low-end tower server can be configured with multiple terabytes of RAM.

E.G. https://www.dell.com/en-us/work/shop/povw/poweredge-t640#tec...

This is a strawman argument. Publishing the code/weights is not mutually exclusive to providing an API

>This is a strawman argument. Publishing the code/weights is not mutually exclusive to providing an API

You're misinterpreting my comment. I'm directly addressing this fragment by the gp: >which can be downloaded, and accessed as a library,

I'm not making any moral ideology statements about the model's "openness", "transparency", or "intellectual property".

As a person very interested in playing with something like GPT-3, I'm talking about practical concerns of even running the model. Some type of cloud API access let's me run experiments today. Hopefully the API cost is reasonable or free with limits. I believe that's true of most researchers because they can't afford the hardware in the near future to run a GPT-3-size model as local library.

> Hopefully the API cost is reasonable or free with limits

If the model was open source you could have an API market where providers competed to build the most economic service just like virtual machine companies do with Linux. If there is only one API then everyone is stuck with them and just has to hope they don't change the prices.

There is practical concern for researchers within the possibility that what costs you $0.05 to run today will cost you $500 tomorrow (see Google Maps API for when this actually happened).

It seems like quite a simple calculus to me: if ever the cost of reasonably using the API exceeds the cost of buying and operating a machine with 350GB RAM, then people will switch to the latter. Either way, I don't see how adding a new option could make anyone's situation worse.

> the cost of reasonably using the API exceeds the cost of buying and operating a machine with 350GB RAM

Capex and opex of these two are quite different.

What's wrong with both? A true democratisation of this kind of model would involve both an offline model for those with the resources to support such a thing as well as many different people hosting them and allowing access for a price via an API for those without those high end resources.

I think the issue people have with centralised cloud APIs for this sort of thing is that there's still a single gatekeeper with their finger on the off switch. In my opinion, instead of throwing out cloud APIs altogether a better scenario would be many gatekeepers with a deliberate diversity of socio-political backgrounds.

If OpenAI made the weights and the NN configuration freely available,I am sure that several organizations would offer cheap (or even free with strict rate limits) API access. And users would also be able to run the model locally, if they can afford the hardware.

This is a common misconception. GPT-3 was trained using a 300B token (~300gb) subset of common-crawl and friends. The model is larger than the dataset.

Reply on news.ycombinator.com