Hacker News Reader

Comment by kelseyfrog | Hacker News Reader

A 5090 has a peak theoretical limit of GenAI 3356TOPS. So we're "already" an order of magnitude greater than what was considered enough for AGI. One question is, "What happened here?" Was the original estimate wrong? Have we notfound the "right" algorithm yet? Something else?

View on HN

"We haven't found the 'right' algorithm yet." seems like the obvious answer, but the numbers in the paper all make sense and I'm interested in some more exotic explanations why it could actually be some orders of magnitude more than a 5090.

Although that's not looking at memory, and I am also interested in some explanation of what... a 5090 has 32GB which, a human brain has more like a petabyte of memory assuming 1 byte/synapse. Which is to say 1 million GB in which case even a large cluster of H100s has an absurd amount of TOPS but nowhere near enough high-speed memory.

We are constantly learning (updating the network) in addition to inference. Quite possibly that our brains allocate more resources to learning than to inference.

Perhaps AI companies don’t know how to run continuous learning on their models:

* it’s unrealistic to do it for one big model because it will instantly start shifting in an unknown direction

* they can’t make millions of clones of their model, run them separately and set them free like it happens with humans

> possibly that our brains allocate more resources to learning than to inference

It's likely in brain inference is learning. If you want a technical analog it's like a conversation in LLMs. Previous tokens do affect the currently generated. I.e. it's inference time learning, well known and widely used.

Nature needed 3.5 billion years to work it out, and we're going to solve it in a few decades?

It depends on where we draw the starting line. We're already at parity with 3.5BYA to 541Mya because no neurons existed in that duration. Only more recently, in the Cambrian, do we have evidence that voltage gated potassium signaling evolved[1].

That changes the calculus likely very little, but it feels more accurate.

1. https://www.cell.com/current-biology/pdf/S0960-9822(16)30489...

I know it’s a silly question to begin with, but if you analyze it seriously, you’d want to at most compare human intelligence->superintelligence with the 20 million years between the first homidinae and homo (and even that is probably too large for some folks to compare with).

One could even argue you should only compare it back to the discovery of writing or similar.

That's not an argument. Nature never worked out going into space, yet we solved it in a few decades.

It worked out flying though, millions of years before we did and we still don't do it as well. We can't even do walking as well as nature did.

Walking is easy compared to elbows, fingers and thumbs. It’s just falling over in a controlled fashion. I hear at least one company in Boston figured it out.

Anyway, humanoid robots should be big in the next 10-20 years. The compute, the batteries, the algorithms are all coming together.

We do flying better. If you adjust for our body weight, a modern airliner uses less energy per traveller mile than your average migratory bird. And the airliner goes much faster.

Yes but that's "in a few decades" ON TOP of millions of years.

If I had to give an estimate, I would consider less the time taken to date, but the current state of our knowledge of how the brain works, and how it has grown in the last decades. There is almost nothing that we know so little about as the human brain, how thoughts are represented, modern imaging techniques notwithstanding.

> Yes but that's "in a few decades" ON TOP of millions of years.

If that's the bar, then anything else can fit in "a few decades", since that also rests "ON TOP of millions of years".

One could argue nature solved it by evolving homo sapiens.

Reply on news.ycombinator.com