Hacker News Reader

Comment by smartmic | Hacker News Reader

I broadly agree with that. Repeated training with self-generated data is the technological equivalent of incest and can lead to nothing good.

View on HN

With a mixture of experts approach it’s not incest, we intentionally split the training data to add diversity. Incest is bad because the errors compound on themselves, but this isn’t a problem when the AI parents don’t share the same DNA (problem space.)

I see it this way, maybe there’s no new ‘knowledge’ but the AI can apply our collective knowledge better than we can. Within that set of knowledge there are surely discoveries never yet realized based on by the fusion of ideas.

Pre-AI we rely on individuals Einstein, Bohr, and Oppenheimer and their associations and studies of each others’ work. With AI, we can fuse the corpus of scientific discoveries into a single entity that we each can communicate with. Maybe the AI lacks the spark of creativity needed to make discovery, but put today’s Einstein in front of it and what would he ask? How much boost would it give him?

Einstein said - “I have no special talent. I am only passionately curious.”

Humans train on self-generated data all the time, but compared to today's LLMs, humans have superior reasoning ability, which enables them to separate good from bad data. Humans can confirm many things for themselves if need be and our social hierarchies crystallize authority figures which perform curation for others. On top of that humans have a lifetime of experience with the real world, while LLMs rely only on reports of what the real world is like.

All in all LLMs being more gullible than your average 6 year old is really not helping. Humans will have to (continue to) curate the data we train LLMs on.

This view of people seems… idealistic, given the number of people who uncritically look to Facebook or Reddit for information (and then happily parrot whatever memes they find) and the number of LLMs that do the same.

I think people are substantially worse at figuring out what is “good” data, and I think a huge number of people are/are starting to nominate LLMs as those “authority figures”. And, given that, I think there is a low ceiling for the creators of these tools to clear to make them appealing to users.

> a huge number of people are starting to nominate LLMs as those "authority figures"

When LLMs started to go mainstream I said I was not afraid of AI, but was afraid of (unwise) people with AI. This phenomenon is what I foresaw.

The worst thinkers are the ones most hungry to outsource their decision making.

The fact that LLM output requires good judgement by the recipient to filter the wheat from the chaff, and those with poor judgement skills are most likely to lean on decision tools makes for a potentially volatile situation.

I think I'll restrain from buying into this until much more testing is conducted. There is information out there suggesting training on synthetic data is at least on par but cheaper.

On par at doing what?

https://huggingface.co/blog/synthetic-data-save-costs

At doing whatever the model is suppose to do.

Training on self generated data is not necessarily such a problem, see how successful alphazero/muzero is when it is only trained on self play.

The key is that you need some kind of external indicator that tells you which generated examples are good and which are bad. In the case of alphazero you get that by simulating games and seeing who wins, in the case of LLMs you will be only taking the generations that are 'successful', e.g. which HN posts get upvoted.

Let's indeed hope that it will still be humans doing most of the voting on HN.

Putting washing machine parts in the laundry, irradiating pre-war steel, etc.

Are we sampling reality, or are we sampling the photograph?

Inbreeding certainly didn't go well for the Habsbergs.

Didn't think that would be a controversial topic!

Only let baby see its own scribbles and scat.

Future world leader advisor.

Ahem, I hope not.

Reply on news.ycombinator.com