Brain FLOPS
geohot.github.io
geohot.github.io
> Assuming each neuron has a fan-out of 1000, aka 1000 MACs, that’s 2000 FLOPS per neuron
Human neurons are not built like we build the artificial neural networks. They may have complex time based behavior ("memory") and dendrites are quite complex, very far away from 2 FLOP per one.
Trying to translate this to FLOPS might be a fun game but I cannot see a real use for any such figure. As for the 3090 that the post mentions, its TFLOPS will drop like a stone once you start deviating from matrix multiplications and im2col towards some more complex structure...
For the sake of this argument, assume information is transmitted and processed in the brain with a binary signal from each neuron, ie. firing or not firing.
From that perspective, it's likely that firing a neuron (which uses a lot more energy than not firing a neuron) is a relatively unlikely event. Ie. if you made a version of the brain which worked in discrete timesteps, the vast vast vast majority of timesteps would not have a given neuron firing.
GPU's don't have such a limitation.
Even the best manmade heatsinks can't dissipate 20 watts and have less than one degree of heating. So I hypothesize that heating is relevant.
We all use simplified models, even outside of Science/Engineering. But the models are, unfortunately, always exhibiting pathological edge cases.
I am always a bit disappointed/sad when I see people like him making such gross oversimplifications leading to grandiose claims with almost nothing to back them up.
There seems to be a "pseudo-polymath" personality type among some engineers. Based on an unexamined but deeply held philosophical belief that engineering concepts and ways of thinking neatly generalize to all fields. The pseudo-polymath engineer is an instant expert in anything he bothers to turn his mind to and subsequently systematize.
Engineers who think they have unique outsider insight into an unrelated field like medicine or cognitive science should tread very carefully. If you can, try to validate your ideas and assumptions with a subject matter expert. Don't underestimate the large stock of knowledge and discourse that already exists in these fields.
Like most positive illusions, the motivation is to forgo truth in favor of power. The difficult, far-away light that borders reality and the abyss, versus, the safe, comfortable, easy to reach thing.
Sometimes it's just fun to squint at a problem a certain way. Sharing that doesn't mean that the person necessarily think it's true. Maybe he just likes to be provocative.
But yet, I think he could do better.
It runs on comparable power and there are oodles of physical events going on in it.
The number of neurons firing does matter, but what matters even more is how they all are connected.
I personally think once we get adequate algorithms and data structures out there, the metric will switch to numbers of lookup operations performed, and floating point operations will just be a multiplicative constant in front of the whole complexity formula.
This is all armchair speculation, but I'd be willing to be my life that a flop is well below the floor of actual computation happening from a single neuron firing.
It seems similar to music/geometric shape. Queries by frequency filtering through the geometry of the neurons activating access to some thought-space.
In computers the address a value is stored at doesn't encode information. Only the value held at the address matters. Not so with neurons. How the connections form and who they are connected to represent compound operations. Like passing a value through separate preconfigured pipelines.
The only things we have for now beyond receptors are CGMP and CAMP and their modulation by phosphodiesterases. That's not enough, not at all. Although it's true the role of phosphodiesterases is understudied.
And it's NOT about size and sCaLaBiLiTy lile deep blue propaganda make people think, the C.elegans has only 150 neurons in its CNS, 300 in total, and no one has built a simulation. It's true that some receptors are lacking and will only be found in the next decade because of underfunding (because of e.g deep blue) but even then it will not run.
Musk certainly is rational - whether that is always apparent or not is another question, and what level of certainty he requires before taking action is another question - what his risk tolerance is - but all things considered he makes more good and rational decisions than not.
I suspect you're in part basing your judgement off of the shallow level of detail that you see in mainstream media or his Tweets, etc vs. ever seeing/reading much of his long-form writing and thinking to be able to understand his rationale; and perhaps also in part it's that your domain expertise don't overlap enough to be able to understand the other's decisions - to reverse energy his actions or shallow words to understand how it flows back to first principles, which he often mentions working from.
https://en.m.wikipedia.org/wiki/IBM_3090
... nevermind.
The Borg may be inevitable.
Yes, but now compare how much time needs a person to leran to recognize a new object vs how much time you need to train a neural network for the same task. You see there is still a problem there, you can't just assume that an AI will be quicker at improving than a human
This might expose the error of the article though. Depending on what you measure by, computers have already beaten humans. But by other measures (now increasingly human biased, like having a cultural conversation), computers are still way behind.
Once we eventually figure out what it takes to implement human-level AI, you can bet there's a whole generation of programmers who do their best to figure out what kinds of weird hardware it's possible to run the thing on.
Same thing as with streaming MP3 and compressed video decoding, DOOM and so on.
I don't buy the overall train of thought, but math doesn't check out, unless he's using numbers I'm not aware of. Commonly accepted number of neurons in the brain: 86B.
86B x 2000 = 172e+12 = 172 PFLOPs.
Where does the assumption of 20W come from?
Note: 2000 kCal per day ~= 95 Watts
Can you explain what is laughable here?
Elon Musk is an other good example, the guy is super smart but also super dumb about a few things, and this is entirely normal and compatible.
We should never look for a messiah.
I notice this reading about Voltaire and how often he is referred to as a "learned" person.
Instead it seems like we just so accept the brain is literally a type of computer that we just haven't figured out the architecture for yet. The storage part is completely trivial like a hard drive so the idea of a "learned" person is equally trivial.
But there is this strange belief that if someone is extremely skilled in a field, his genius can be applied with little efforts and training elsewhere.
You can be a Chess champion and an excellent Chef, but you this is completely unrelated, except maybe for the discipline.
Same with Philosophy and Theoretical Physics, or almost anything.
You can even be a World class Quake player and be mediocre or barely average at CS.
It is a large aggregate of abilities and skills, with some connections between them.
And we also tend to confuse language and rhetorical ability with general intelligence, because this is an ability we all have and need, that is also acting as a proxy intelligence benchmark but I don't think this ability is general enough to be used in this way.
So the fallacy behind the fake polymath is not the idea that being smart makes you better at analyzing information. It's the idea that being smart is a substitute for information and expertise. This mistaken assumption can lead smart engineers to beliefs that subject matter experts consider religious or delusional. Mars colonization is a great example in Elon Musk's case.
Btw most of the general training you mentioned isn't part of STEM and is historically associated with humanities subjects like law and philosophy instead. In STEM you typically work with set epistemic frameworks with unambiguous correct and incorrect answers. In this sense, engineers actually learn the opposite of critical thinking and discourse.
> assuming the brain is drawing 20W
> Assuming each neuron has a fan-out of 1000
> to be fair, they are sparse
Also
* 1 ATP activation is equivalent to one FLOP
* Just having FLOPs is enough to make a human brain
It's idiotic.
This feel like a variation on Betteridge's law of headlines which states that if a headline asks a question, then the answer is "no". In this case if the quote says that 'maybe' something (fanciful) is true, it isn't.
https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
Where does it think it came from? The power supply? Who has absolute power?
If you want to design someone like you unconsciously though, that’s pretty easy - just takes another person and nine months.
Do computers forget on purpose? No, only when we tell them to. Do humans forget? All the time.
I don't know what I don't know.
George Hotz' website (original link) says "the singularity is nearer". Wanna bet that "the singularity" is already among us?
You always needed to be sharp to survive and to create children who do the same. Unless you are a species that doesn't need to create a highly cognitive offspring, for example insects, for whom what you say probably truly applies.