Hacker News Reader

Comment by famouswaffles | Hacker News Reader

>It feels like computers should be able to get much more out of what is already available

I mean why ? It took millions of years of optimization for humanity to get to the competence level they're currently at. If you think you're "starting from scratch", you really aren't. Keep in mind LLMs can use significantly less data (but still a lot) when you're not trying to force feed the sum total of human knowledge.

So should they be able to get more out of it ? or is this par the course for NNs?

View on HN

> If you think you're "starting from scratch", you really aren't.

Our genomes have ~600MB, so where are you claiming that humans transmit millions of years of language optimization between generations?

You could store a pretty gigantic graph using 600MB.

They're not saying we store the weights, they're saying that we store the "architecture" that you overlay the weights on with training.

That 600MB is the result of millions of years of optimization. For every human genome that exists today there are many other genomes which were tried and discarded over the years.

It also contains a remarkable amount of compression. Even if you assume that a genome contains a complete template for a human (it likely doesn't) the fact that the compressed version is 600MB doesn't really relate to the unpacked amount of information. Especially since the compression has seen millions of years of optimization.

Because humans with less language data outperform LLMs with more language data.

This either says we need better models, not more data.

Or, the human ability to be multi-modal augments our ability to perform language tasks in which case, we need to pump LLMs with much more image and video input than we currently do.

The point I'm making is that humans do not in fact have "less language data". We're heavily predisposed to learning languages. We don't start with random weights.

GPT has no such predisposition.

Better init and architecture aren't what people think of when they think of "giving models more data" - they mean a larger training set.

You're not making much sense here. The better init comes from training and little else.

GPT needing lots of training data doesn't mean we need a better architecture. You would expect it to have a lot of training because humans have a lot of training too, spanning millions of years..

Init means the distribution of weights prior to training.

Human training begins at birth.

Evolution might result in better architecture and init(inductive biases), but that's a separate thing than training.

> Human training begins at birth.

Is this true? My understanding is that people are born with many pre trained weights. Was the evolutionary convergence of those weights not itself training?

> Was the evolutionary convergence of those weights not itself training?

No, inductive biases are not training.

I'm saying that better models(ie: better inductive biases) or non-language data is needed to advance LLMs and somehow we've arrived at "evolution is training." I'm not sure how that's relevant to the point.

Evolutionary training is not just inductive bias. They're not comparable at all lol.

And the more inductive bias we've shoved into models, the worse they've performed. Transformers have a lot less bias than either RNNs or CNNs and are better for it. Same story with what preceded both.

This is a good time to stop and ask, what point do you think I'm making?

Pretty much all of your assertions in this thread and https://news.ycombinator.com/item?id=37797108 in particular are what we're disagreeing with.

Right, I'm checking that you understand what point I'm making by asking you to reflect it.

Evolution has determined a good architecture. The weight training is then just the final tweak to get everything running smoothly.

No reason beyond compute we couldn't do something similar. Ie find good architectures by evaluating them using multiple random weights, and evolve those archigectures that on average gives the best results.

Then over time add a short training step before evaluating.

You're comparing humans - a multimodal model with billions of years of training epochs - to a unimodal language model that's been around for a few months.

Correct. That's a crucial part if the point I'm making.

Yes but heavily inspired by the billion year old model ?

Let alone no hardware squeeze and boundless energy supply through the food chain. Remember for most of the billions of years we were not human and were low powered

Millions of years of evolutionary computation is a fairly small amount of computational time. LLMs also benefit from decades of neurological computation in that their structure was invented and optimized by humans, which is already orders of magnitude faster than evolution.

Reply on news.ycombinator.com