Hacker News Reader

Comment by CoastalCoder | Hacker News Reader

Out of curiosity, what happens to an LLM if, over time, it's increasingly trained on its own output and/or that of other LLMs?

And has anyone figured out a good way to minimize that when sourcing training data from e.g. Reddit?

View on HN

I read (probably on HN) where someone suggested that pre-LLM data will become the 'low background steel' (pre-Nuclear-age steel with lower levels of internal radiation) of machine learning.

It seems like provenance/traceability of data sources will become important, both for the builders and users of LLMs.

There's a no-longer-very-speculative bit of Sci-Fi waiting here about whole societies "forking" based upon the curation of their training data. Arguably social-media + bots + paid state actors means we're already there.

Isn't "societies forking based upon curation of their training data" practically what culture is?

Now, I agree that having scalable generative AI and high-throughput-high-virality mass media (social networks) brings new unspeakeable horrors like feedback loops to the mix. Interesting times indeed.

can you point to an actual paid state actor. the term seems like mostly bullshit.

Habsburg AI – a system that is so heavily trained on the outputs of other generative AI's that it becomes an inbred mutant, likely with exaggerated, grotesque features

from https://twitter.com/jathansadowski/status/162524580321127219...

Or even, the Zuckerberg of AI in a meta-sorta-way''

This was pretty clever:

>CaligulAI <-- @Rogntuudju

> CaligulAI

I believe it's spelled "Caligulae" because it's first declension.

/Latin-joke

LOVE it

You know what would be funny ;

."Latin Joke Explainer" -- As explained in Latin Man-Splain-Terms."*

EDIT:

(You mastered my joke, and I appreciate it.)

EDIT AGAIN:

UI think we actually just coined a term ; 'Caligul::AI' -- Malicious AI for its own pleasure.

https://news.ycombinator.com/item?id=36319076

Hm, sounds like dog-fooding, but where the 'food' is dog shit. Dog-shitting? Dog-shit-fooding?

Dog breakfasting

dogging

Dogenshittification

The Poopboros.

AI Centipede

I certainly have no idea but interesting to think about! One could argue that humans are trained on their own output so presumably interesting things could arise, especially if it's not a totally closed loop (LLM#1 -> some human curation -> LLM#2).

> humans are trained on their own output

And it seems we're emulating that same cycle. Humans are trained not only on our own output, but also take changes from our environment. So in this sense LLM's environment would be human input.

the problem with the current tech is that the solid part of knowledge isn't separate from the fluid parts. You need to fundamentally understand that human knowledge in many generations was purposely fixed, by books, scholars, etc.

Until that occurs, it's really just a miasma of bullshit.

Lately I have been trying a new LLM called Falcon that was supposedly created in the UAE but it was mostly trained on conversations with ChatGPT... the result is that if you ask Falcon who created it, it will happily say OpenAI. I found that really amusing.

The instruct trained variation, you mean.

I believe that performance degrades, as the models end up training on data that is more homogenized and less diverse when compared to original data.

At least according to this paper (I think, FYI I am not a expert) https://arxiv.org/abs/2305.17493

"Model Collapse"

https://venturebeat.com/ai/the-ai-feedback-loop-researchers-...

I read a research paper which says LLMs don’t actually improve by training on the outputs of other LLMs. They only gain the ability to answer the specific questions included in the LLM output training dataset. Their reasoning capabilities etc. don’t improve.

I’m probably wrong. See the Orca model which came out recently.

Possibly such data will be used in a different way. If it's embedded in a social network, engagement (votes, retweets,wwillhstecer) will act as a sort of crowd sourced reinforcement learning source instead of being direct part of the training set.

Perhaps the source training data will shift to transcripts or subtitles of podcasts, cable television, TV shows, and other sources that haven't (yet) been polluted.

Combined with pre-LLM data sets.

Training is becoming multi-modal anyway and will need grounding in physical reality so human-generated text will most likely become a smaller fraction of the data.

Hopefully some of this is democratized away by humans voting on quality of output directly and/or indirectly?

Have you seen Multiplicity?

"Hey Steve, come on up, I'm spittin on bugs"

It can reinforce its own biases for one

You’ve heard it at a concert when the microphone picks up sound from the speakers and a positive feedback loop ensues. A lot of screeching.

Context drift.

Reply on news.ycombinator.com