Hacker News Reader

Comment by nojvek | Hacker News Reader

Wow. This is amazing. Although I wish they documented how much compute and data was used to get these results.

I absolutely believe we'll crack the fundamental principles of intelligence in our lifetimes. We now have capability to process all public data available on internet (of wikipedia is a huge chunk). We have so many cameras and microphones (one in each pocket).

It's also scary to think if it goes wrong (the great filter for fermi paradox). However I'm optimistic.

The brain only uses 20 watts of power to do all its magic. The entire human body is built from 700MB of data in the DNA. The fundamental principles of intelligence is within reach if we look from that perspective.

Right now GPT3 and DALL-E seem to be using an insane amount of computation to achieve what they are doing. My prediction is that in 2050, we'll have pretty good intelligence in our phones that has deep understanding (language and visual) of the world around us.

View on HN

> Although I wish they documented how much compute and data was used to get these results.

I'm hearing astonishing numbers, in the tens of megawatts range for training these billion-parameter models.

And I wish they showed us all the rejected images. If those images (like the snail harp) were the FIRST pass of the release candidate model.... wow... but how much curating did they do?

EDIT: Units. Derp.

> tens of mega-joule range

do you mean tera-joules?

A hundred megajoules is about three bucks at 10 cents per kwh.

I routinely do giga-joule level computations using just a rack of computers in my garage, they're no big deal.

metawatts?

Joules is a unit of total computation, watts is a unit of computation rate. :P

Metawatt is a unit for rates of speculation, uninformed by multiplication, about AI energy usage.

Oops, *mega, obviously! But I did mean watts.

MWh is what you'd want.

I was thinking power consumption. Probably a mis-correction anyway.

> in the tens of megawatts

But for how long? 1 second, 1 hour, 1 month? The energy matters more than the power.

Maybe a nitpick, but there's a difference between energy consumption during training and inference. If you want to talk about the energy necessary to train a human brain, it involves years of that 20W power consumption. What is the power consumption for inference time for Dall-E?

Not to mention the billions of years spent in the evolutionary pipeline.

Well, if you want to go that route, you'd need to count all the energy spent to build computers of all kind since the 50s, and also all the energy spent to sustain the lives of people working on AI. And, well, all the millions of years spent in the evolutionary pipeline before these people where born ;)

I did some kick napkin math for Lee Sedol vs AlphaGo a few weeks ago: https://news.ycombinator.com/item?id=25493358

Wow. Good to know that entire lifetime energy consumption is 50 MWh. At $0.1/KWh, you’re looking at $5000 in equivalent electric energy consumption over entire lifetime of a human being.

The brain uses 20W of power. For a life time of ~80 years, that is 14MWh of energy usage. Suppose we say the brain trains for the first 25 years then that is 4.38 MWh. Equivalent electric energy consumption is only at $438.

So yeah, the brain is quite efficient both in hardware and software.

That's still really surprising I think. I would have imagined that it would have been off by many orders of magnitude. The fact that these models are within a factor of 3-10 of the human consumption is pretty impressive.

That being said, these models are training only for very specific tasks whereas obviously the human brain is far more sophisticated in terms of its capabilities.

Bear in mind that only a tiny fraction of the energy spent by Sedol's brain during his whole life was dedicated to learning Go. Even while playing Go, a human neural system spends a big part of its energy doing mundane stuff like moving your hand, standing up, deciphering the inputs coming from the eyes and every other sensitive body parts and subconsciously processing it (my back hurts I need to change posture, my opponent smells well, the light in the room is too bright, etc.). Interestingly enough, doing most of these things is also a big challenge for IA today.

Do you think we’ll crack the principles, or do you think we’ll just have very powerful models without really knowing what makes them clever?

My bet is understanding the fundamental principles. Like building an airbus plane or starship requires fundamental understanding of aerodynamic principles, chemistry, materials and physics.

DNNs will definitely not get us there in their current form.

I am very curious to see if concepts from cognitive science and theoretical linguistics (like the Language of Thought paradigm as a framework of cognition or the Merge function as a fundamental cognitive operation) will be applied to machine learning. They seem to be some of the best candidates for the fundamental principles of cognition.

They don't need to solve the problem of reasoning, they only need to simulate reasoning well enough.

They are getting pretty good, people already have to "try" a little bit to find examples where GPT-3 or DALL-E are wrong. Give it a few more billion parameters and training data, and GPT-10 might still be as dumb as GPT-3 but it'll be impossible/irrelevant to prove.

> The entire human body is built from 700MB of data in the DNA.

I think this notion is misleading. It doesn't relate to the ease of simulation it on our current computers. You'll need a quantum computer to emulate this ROM in anything like realtime.

The DNA program was optimised to execute in an environment offering quantum tunnelling, multi component chemical reactions etc.

My point wasn’t to emulate the DNA in a virtual machine. The point was that between humans and chimps (our closest DNA relatives) we’re 99% same to them. So high order intelligence that gives rise to written language and tool building is somewhere in that 700MB of DNA code. And 1% of that (just 7MB) is responsible for creating the smartest intelligence we know (humans).

In that sense intelligence artchitecture isn’t very complicated. The uniformity of isocortex which we have the most relative to our brain size compared to any animal says we ought to replicate its behavior in a machine.

The isocortex/neocortex is where the gold is. It’s very uniform when seen under microscopes. Brain cells from one region can be put in another region and they work just fine. All of ^ says intelligence is some recursive architecture of information processing. That’s why I’m optimistic we’ll crack it.

I think what the parent was saying is important though, that 700mb of "data" isn't complete. It's basically just really really good compression that requires the runtime of our universe to work properly. The way proteins form and interact, the way physics works, etc are all requirements for that DNA to be able to realize itself as complex thinking human beings.

Yes you rephrased that very well.

If your plan to build intelligence is by copying how nature does it, well then you'll need to build a "nature" runtime that can emulate the universe. You can either do that slowly or inaccurately

> I absolutely believe we'll crack the fundamental principles of intelligence in our lifetimes.

I tend to agree. However this looks a lot like the beginning of the end for the human race as well. Perhaps we are really just a statistical approximation device.

Yeah why i mentioned the great filter of Fermi paradox.

I also believe humans in our current species form won’t become a space bearing species. We’re pretty awful as space travelers.

It is very likely that we’ll have robots with human like intelligence, sensor and motor capabilities sent as probes to other planets to explore and carry on the human story.

But future is hard to predict. I do know that if the intelligence algorithm is only in the hands of Google and Facebook, we are doomed. This is a thing that ought to be open source and equally beneficial to everyone.

Reply on news.ycombinator.com