Hacker News Reader

Comment by milleramp | Hacker News Reader

Is there a term for this where outputs are eventually fed back into training, at a ever increasing rate? I propose the Malkovich effect.

View on HN

One researcher has called the phenomenon "Habsburg AI", likening it to inbreeding.

https://www.axios.com/2023/08/28/ai-content-flood-model-coll...

"Es war sehr schön, es hat mich sehr gefreut."

> Is there a term for this where outputs are eventually fed back into training, at a ever increasing rate? I propose the Malkovich effect.

Model collapse?

https://en.wikipedia.org/wiki/Model_collapse

https://www.theregister.com/2024/01/26/what_is_model_collaps...

All areas of systems theory [0] whether in the cybernetics [1] of Norbert Wiener [2] or in complex real systems like those Meadows [3] and Forrester [4] studied, the divergence from equilibrium that occurs in positive feedback loops is both well studied, yet strangely hard to understand.

We don't know quite what is going happen, because selective amplification of different components will lead to different effects. It's unpredictable. Sometimes we get oscillation. Sometimes we get chaos or noisy behaviour. This relates to "chaos theory" [5].

sometimes we get a discontinuity, either an instant collapse or a "pole" that shoots off to infinity. Sometimes we get all these things in some strange sequence, although the latter two are usually unrecoverable.

The biological system I think is closest to what we're doing with AI is BSE (Bovine spongiform encephalopathy) [6] when we fed dead cow bones back to living farm animals which selected for prions.

[0] https://en.wikipedia.org/wiki/Systems_theory

[1] https://en.wikipedia.org/wiki/Cybernetics

[2] https://en.wikipedia.org/wiki/Norbert_Wiener

[3] https://en.wikipedia.org/wiki/Donella_Meadows

[4] https://en.wikipedia.org/wiki/Jay_Wright_Forrester

[5] https://en.wikipedia.org/wiki/Chaos_theory

[6] https://en.wikipedia.org/wiki/Bovine_spongiform_encephalopat...

Funny how they don't mention the solution: to have AIs actively participate in the real world. I mean, it's hardly what people here want to see happen, but that will solve it.

Current LLMs cannot "actively participate in the real world" as humans do because they cannot actively learn from their interaction with the real world. Their weights are fixed. Their further training and finetuning is mediated by external mechanisms. It has nothing to do with what "people here" want.

Sure, though it seems there are a number of near term paths for improvement. Short and long term memory mechanisms would go a long way towards active learning. Fine tuning could be iteratively performed through mechanisms like low rank adaptation. It's been hypothesized that this is one of the purposes of sleep, and humans are consolidating memories during dreams.

Not true. There's plenty of easy ways to update the weights if you want to do that.

You could for example DPO based on another LLM's evalution of how a conversation is going, or you could use whether the conversation keeps going.

>>>"You could for example DPO based on another LLM's evalution of how a conversation is going,"

You can update the weights based on a single point of data (response was bad) but you probably can't usefully update a model that way.

I can't seem to find anyone actually usefully trying this. Do you know of any data?

No but Google (for example) recommends fine-tuning based on at least 500 examples of question response pairs. DPO as far as I know requires a good and bad example. It's not a technique based on just saying "ai response is bad".

But that's practically what's being done in supervised learning.

If you would isolate a child in front of a CCTV screen, it also wouldn't magically learn from the images what's our definition of a tree, a bush, a bike, a motorcycle, a car. Someone would have to take it aside and explain it first.

In AI you just don't take it aside, you build a new child.

Supervised learning is "Hello lil CCTV model. Here's a bucket of some 'world' data, labeled in a way for you to ingest. Here, some images of T-R-E-E-S...".

How?

By releasing them from being confined to their own farts.

That frames it like ML-models are caged intelligent beings which would prosper if only someone would set them free...

You know, actual "intelligent beings" are also next-token-predictors that are caged in a mechanism that forces them to give certain outputs ...

I've been coming back to this comment and I'm not sure what you mean exactly.

The brain is trapped in the body, and forced to meet its needs (e.g. regulate hormone levels).

The brain and body are trapped in civilization, and forced to produce "outputs" it considers acceptable.

All three, in turn, are trapped in natural selection.

"Habsburg AI" - https://x.com/jathansadowski/status/1625245803211272194

MAD: Model Autophagy Disorder (https://arxiv.org/abs/2307.01850 for the paper that introduced the term)

Brings to mind "mad cow disease" [0], caused by cows eating other cows (through meat-and-bone meal nutritional supplements in their feed)

0: https://en.wikipedia.org/wiki/Bovine_spongiform_encephalopat...

You got my vote. What a great name for AI dregs feeding back into AI training to turn out even more bizarre dregs.

For those unaware of the reference, movie: "Being John Malkovich".

A negative/positive feedback loop, depending on what you consider negative or positive.

I'd go negative, with the y axis being "useful". Negative feedback loops all go to zero.

Language processing differs from games like chess because with chess you can definite a clear metric of quality. If you can define such a metric, AI models can learn from themselves and have a near-infinite positive feedback loop. See AlphaGo etc.

There's an upper bound for the sum total of human text output on the Internet. when large language models run out of training data, we can only train on large language outputs.

Positive feedback loop ? With the quality that it slowly reduces "original" content ?

Reader's Digest Effect?

I suspect we will need several different terms to describe these inbred AIs (hey that's not a bod one). Besides the post 2021 AIs being trained on polluted data, there some models right now that are trained on output from other AI (on purpose).

The Cybernetic Centipede?

What's that? Did you actually mean the Milankovitch effect ?

I think it’s a reference to the movie, being John malkovitch

The AI model equivalent of Kessler syndrome

GIGA - Garbage In Garbage Out

Garbage In, Garbage AI

I laughed!

> Is there a term for this where outputs are eventually fed back into training, at a ever increasing rate?

Consciousness

Reply on news.ycombinator.com