Hacker News Reader

Comment by SamBam | Hacker News Reader

I was thinking about this when reading an article on HN a couple weeks ago, "LLMs Aren’t Just “Trained On the Internet” Anymore" [1] (HN discussion [2]).

The thesis was that it won't matter if LLMs exhaust the available corpus of text that humanity has generated thus far, because it will always be easy to pay humans to create an inexhaustible supply of more text.

The article suggested that it was all going to be PhDs, lawyers, and poets creating this training data, but that all sounded like hand-wavy BS to me. I assume that, if more text is needed to train LLMs in the future, the vast majority of it is going to be pennies-on-the-page, mechanical turk-type workers. And the LLMs are going to reflect that kind of input.

1. https://allenpike.com/2024/llms-trained-on-internet

2. https://news.ycombinator.com/item?id=40549021

View on HN

The issue is that as AI gets better and more accessible, more and more people will use AI to help them with MTurk-style work that requires writing text[0]. The only sure way to get human-generated text will be to either have a person go to an office (expensive), or more likely just force all workers to install 5 cameras surveilling their every move with some algorithm that may fire a few people for taking a bathroom break. Or, more easily, just scrape more data

[0] https://techcrunch.com/2023/06/14/mechanical-turk-workers-ar...

>just scrape more data…

…that was generated by LLMs. So no you’re back to problem of getting data generated by humans.

In which case, why not just have humans write the text we need instead of firing all our writers in the first place?

R/AskReddit has prompt posts that seem to tap into people’s opinions about current events and how people feel about various things. My best guess is that it’s LLM trainers tapping into the oblivious masses.

Even before the rise of LLMs, that sub was heavily used to create listicles (“10 worst things about foo”).

And of course those LLM trainers have no clue how many of the responses come from AI-generated bots. Ouroborus of chat.

If corporations can simply augment some generated text with the cheapest human labor available, without audiences noticing or caring so much that it affects their bottom line, they will do so. Corporations exist to optimize for cost/profit, nothing more.

> Corporations exist to optimize for cost/profit, nothing more

But LLMs are not only used in corporate settings. Cutting the bottom line in medical professions, case law, academic journals, etc. may seem enticing, but their audiences will definitely bear the brunt of the loss of specialist diagnoses, case work and peer/lit reviews.

Agreed, their audiences will definitely bear the brunt of those losses, but they will still absolutely try, and your average person has very little (if any) control over this.

(Which is very very scary - at the very least: Law, medicine ...

... how many (and to what extent) are today's students/trainees and future practicing pros already passing classes that they would have otherwise failed, undetected -for example- and undermining future professional services which we will all avail ourselves of?

Ease of use?

Like why pay more for AWS when you can host things locallly. People just can't be bothered with the admin

Latency

the Internet has generated vast digital language data in forums and publications, ads and opinion writing.. this is discussed in detail already.. there is concern about training on low-quality thinking and writing, and very much concern about using LLM output to train new LLM.. there is so much content needed for training, no group of humans can review it all. "bias" is the first word they use but really there are endless flaws that are possible.

Don't we also eventually run into a situation where so much of the online content is AI generated that LLMs are being trained mostly on their own data. In that situation, it seems we could get stuck in a self-reinforcing loop rather than branching out with creativity and ingenuity, because even if such content still exists out there it risks being drowned out.

Unless there's huge leverage specific text/topics, I can't imagine MTurk anything will make a dent in the amount of text available for ML training. Your best bet imo is to somehow detect and filter out enough LLM generated text from reddit or wherever to not poison yourself.

Data quality matters for training, at least as much as quantity.

A possible counterpoint here is The Unreasonable Effectiveness of Data by Halevy, Norvig and Pereira.

https://static.googleusercontent.com/media/research.google.c...

Say what you will about the merits of this paper, it's hard to deny that it has been influential within the deep learning community.

That's a bit outdated. At the scale LLMs are trained, quality starts to matter again.

The pay that workers earn suggest otherwise

There's AI annotation work paying into the $100+/hour range. Not nearly as much as at the lower rates, but the amount in the $20-$40/hour range might surprise you.

(been offered contracts, have done some on the top end of the market, have access to data on some of the volume on the lower end)

Can’t found anything near that, $40 for expert roles is the most common I find. I worked with one of these vendors, and the working condition is so horrible that people who are actual experts would not tolerate

Reply on news.ycombinator.com