Hacker News Reader

Yeah I posted this in another thread but your claim is simply untrue.

There is little to no need for anyone to scrape the web themselves, CommonCrawl does it for free and provides access to all that data for free, they are a charity similar to The Internet Archive:

https://commoncrawl.org/

As for using that data to train a model, I've done it at my own company and it's not that expensive, certainly not the astronomically expensive figures people seem to think it is. Maybe a hobbyist can't do it for themselves, but my small company managed to get pretty good results training on our existing hardware. It's hard to give concrete figures because we manage our own hardware instead of using AWS, but all in we're talking on the order of 100k, as opposed to what I've heard people on HN think it costs, as if it's a $10 million+ enterprise.

Yeah and the model will be useless when the courts determine it's illegal to use it commercially without having properly licensed the data.

Remember... Bits have color and AI models are a rainbow.

“The courts” don’t have rules which apply globally for this sort of thing. My understanding is that even if the models were made illegal to host in America, there’s no shortage of other countries much looser copyright laws where one could hypothetically host such a thing.

There are clearnet sites which host entire movies and games and there have been since the dawn of the internet, wouldn’t you expect those would be bigger targets to go after if the powers were there? All that to say, I’m not bullish on the future global leverage of American copyright law, especially when it comes to AI models.

But courts have jurisdiction over their consumer markets, if a court in the US says it's illegal to use non-licenced data for training models and any product doing so can't be used/sold in the US you have lost access to a massive consumer market.

Case in point: GDPR, you can vacuum as much private data as you wish from non-EU citizens, the moment you touch EU-citizen data you must abide by the GDPR or you don't have access to a market of 400+ million of some of the wealthiest countries in the world.

You don't need global jurisdiction when your economical power is large enough.

Which leaves the citizens in countries without large enough economical power at the mercy of those corporations. Nothing new, you will say.

Big corp will always have more data. Even just the web, search engine have a big head start. Then there are "private" data that big provider use for "improving our product": emails, word documents, chat messages, non public social media posts,...

That's a useful number. Just yesterday we had an announcement of an open source system you can train with one GPU on a desktop for simple models. That could usefully use 16 or 24 GPUs, but it's not like you need a whole data center. It doesn't look like this is an industry with excessive monopoly potential.

The real issue for machine learning is that it lets you build systems that optimize for some metric. That's what corporations do. I've been pointing out lately that the ethical problems for AI in the near term look very much like those for corporations.

I’ve been thinking along the same lines as what you seem to have done. I looked into the CommonCrawl data and also the Pile and I’m not sure they are as large as people think. We also manage some of our own hardware including GPUs. It looked to me that with maybe .5 PB decent storage and 8 high end gpus it could be possible to train a decent LLM in a few months, how does that estimate compare to your experience?

Something I’ve been pondering lately: with the release of all of these content generating language models lately, what will happen to the “scrape and train” process going forward? Aren’t we about to realize an infinite loop problem basically where the training sets will be full of ai generated content that isn’t actually useful for training?

Seems like we almost need pre-ai scraped datasets, almost like how it’s sometimes very useful to have metals that were created before the first atom bombs (I don’t know too much about this, just that there’s a definite land in the radiated sand around that timeframe).

I wonder how correct this is. There are two growth factors here. One is the amount of data required to train a premium model is increasing. The second is that the amount of data being generated is increasing.

I think the future you described is possible. I also think it's possible that corporations are overpaying for data that will become stale after several years and new fresh data is generated in such great quantity/quality that access to it is not capital intensive.

high quality modern 'AI' tools requires a metric assload of data

There seems to be a lot of different ideas about 'quality'. A lot of people want a better-than-search tool that knows vast amounts and can be assigned tasks. I'm personally much more interested in a reasoning machine, I don't need it to include the entire contents of the New York Public Library.

Eh not really, we aren’t far off from AI being able to use browsers just like humans and operate in the world like us. There are no reasonable laws that will be able to prevent this

Yeah there are... There is no fair use for AI. Fair use is about human rights to their own thoughts. AI does not have that.

The law is simple... You have to have a license to the data to train a model with that data. If you don't... Get ready to be sued.

Becareful of regulatory capture tho. If the one with AI headstart can capture a huge swath of value, said value can be used to lobby and bribe legislators or the executives of the government.

There's no good way to put the AI genie back in the bottle, or make sure it will not create catastrophic undesired consequences. Only way is forward, even if there is no good path in sight.

> The law is simple... You have to have a license to the data to train a model with that data. If you don't... Get ready to be sued.

This won't be the case everywhere. UK has actually already explicitly ruled the opposite

> There are no reasonable laws that will be able to prevent this

any law can be legislated, if it can be put down as text.

It may be considered unreasonable, but if the majority of people _do_ agree, then it could be done.

> operate in the world like us

To operate in the real world like humans you need to have motivation. What is an AI motivated by?

Its goal function.

Sure, but what would it be?