Hacker News Reader

Comment by skybrian | Hacker News Reader

The future is not known to us. But given how inefficient machine learning seems to be, algorithmic efficiency improvements may keep the scaling going for a while? Maybe that's not a "major breakthrough" but it's improvement nonetheless.

It's also going to take a while to learn to use the new toys we already have.

View on HN

> given how inefficient machine learning seems to be

We emulate neurons mathematically but it is possible to build efficient analog circuits that emulate them physically.

I doubt any biological system can ever learn all the information gpt4 has learned. The gpt4 learning may not have been power efficient but neither were the first airplanes compared to birds yet today biological flight is rather limited compared to flight that uses technology.

Ever listen to Geoffrey Hinton speak about back propagation vs what biological systems use? Do you think he is wrong about this?

He isn't wrong, but with current GPU hardware back propagation is actually the superior algorithm compared to his latest forward forward algorithm. Swapping the training algorithm doesn't get around the fact that you need to perform lots and lots of forward passes and those need an insane amount of memory bandwidth. Back propagation isn't a significant bottleneck in computers and also not in terms of memory capacity. So the only benefit of the biologically plausible forward forward algorithm is that you could run it on a digital or analog NPU, but with Ryzen AI coming in Strix Point, every high end laptop is going to have 50 TOPS of AI compute and 200GB/s of memory bandwidth. Nothing except large datacenter GPUs or a hypothetical 5090 with 32 GB would be competitive against that anytime soon. Anything smaller and you will have to buy several 3090 on eBay for like $900 a pop. Analog circuits are far off for now.

> I doubt any biological system can ever learn all the information gpt4 has learned

The fact that human brains can become competent at language with training datasets with a size of a tiny fraction of common crawl points to how much more efficient they are

Until you realize that ChatGPT is also fluent in 20 other languages by including minimal data into the training dataset.

Minimal as in the amount a person needs to learn the language or minimal as in tiny if compared to English in common crawl but still orders of magnitude more than a person goes through for infancy language acquisition?

I think you know the difference.

It’s anecdotal, but here [1] is an example of AI learning to translate a new language faster than a human could learn to do it, from a similar amount of data, using Claude 3 Opus.

Translating a language isn’t quite the same as understanding it, but this is still impressively data-efficient.

[1] https://twitter.com/hahahahohohe/status/1765088860592394250

Did you read the "added context"?

"The author has later responded and apologized a flawed / biased methodology which led them to believe that Opus has no prior knowledge of Circassian (i.e. wasn't trained on it)

Apparently it was, and is able to speak Circassian just not perfectly"

I didn't see that! Thanks!

Bro you are proven wrong. Baby FOV camera footage as the training dataset of LLM, the result is not bad.

https://www.nyu.edu/about/news-publications/news/2024/februa...

The result is bad, that is just an image recognition model we know those can be trained with relatively little words and just a bunch of images, the model didn't learn grammar or sentences or relationships like a baby would. The only models we have that can do grammar and sentences and relationships are LLMs with massive amounts of data.

I'd love to see more algorithmic efficiency in ML. But hasn't ML been going in the other direction for at least a decade now? It seems to me that it's been going aggressively towards brute force algorithms. Specifically, figuring out how to do as much matrix multiplication as possible.

Among other things, there seems to be progress in fixing attention to scale better.

Reply on news.ycombinator.com