Hacker News Reader

Comment by tomohelix | Hacker News Reader

Aren't we running out of text to train AI? You need much more data to train a smaller LLM to match a larger LLM performance. Considering how everybody is starting to clamp down on their user's data, we might have issues getting enough to train something on the scale of PaLM to reach optimal performance.

On that note, everyone is probably salivating over Discord's data. So so much chat based and detailed conversations, all neatly organized into groups...

View on HN

I am wondering how much value social network data like Twitter or Discord really has when compared to something like Wikipedia or textbooks.

It is mostly unformatted, contains slang and is often far from the truth. Is this data really all that useful for training an LLM?

Social media has more idiosyncratic and human-esque text.

People already complain that ChatGPT is too formal and verbose, as an AI language model.

That's probably more because of RLHF though, they've optimised for certain kind of responses rather than simple model loss on internet text.

Unformatted is irrelevant (or a benefit).

That is contains slang is a huge benefit - how else will it learn slang?

That is often far from the truth is of mixed value. Firstly it is very unclear that a LLM is the best method for holding facts. Secondly, the "untrue" data at least gives the LLM an idea of what could be true given a context, which lets it learn a better model of the world.

For example, if it gets a lot of text wrongly claiming that "Winston Churchill is the 40th President of the United States" it will add to the evidence that "The President of the United States" and "Winston Churchill" are both in the class of "people".

This is opposed to nonsense text like "A carpet States President United" (which is just noise).

Why do we want the LLMs to understand slang ? Will it use slang to cure diseases ?

Who is this "we" you speak of? Because the people using them to generate lyrics certainly want slang.

But yes, there are plenty of scenarios where knowledge of slang could cure diseases.

Consider mental health, where one of the main effective interventions is diarying. If a LLM can understand the slang in a diary then it is certainly possible it could intervene successfully.

Sounds like a joke.

It's probably more training-compute intensive, but they can do drop-out, right? The strategy they used for ImageNet recognition, when they were using supervised learning and training data was scarse.

Dropout is one strategy for regularization but doesn't guarantee avoiding overfitting, especially now that modern AI models generalize much better than they did during the ImageNet days. Many of the big LLMs use a dropout of 0.1 though.

Using lots of low quality data is counterproductive. Unless you're specifically trying to imitate flawed humans.

Yep we are running out of text data, see https://doi.org/10.1098/rsos.221454

Reply on news.ycombinator.com