Hacker News Reader

Ignoring for the moment that transistors and synapses are very different in their function, the current in a CPU transistor is in the milliampere range, whereas in the ion channels of a synapse it is in the picoampere range. The voltage differs by roughly a factor of ten. So the wattage differs by a factor of 10^10.

One important reason for the difference in current is that transistors need to reliably switch between two reliably distinguishable states, which requires a comparatively high current, whereas synapses are very analog in nature. It may not be possible to reach the brain’s efficiency with deterministic binary logic.

>the current in a CPU transistor is in the milliampere range

? you sure about that? in a single transistor? over what time period, more than nanoseconds? milliamps is huge, and there are millions of transistors on a single chip these days, and with voltage drops of ... 3V? .7V? you're talking major power. FETs should be operating on field more than flow, though there is some capacitive charge/discharge.

Single transistors in modern processes switch currents orders of magnitude lower than milliamps. More like micro- to picoamps. There's leakage to account for too as features get smaller and smaller due to tunneling and other effects, but still in aggregate the current per transistor is tiny.

Also the transistors are working at 1V or lower, but as you say they are FETs and don't have the same Vbe drop as a BJT.

You are right, I mixed this up. If you take a CPU running at 100 W with 10 billion transistors (not quite realistically assumed to all be wired in parallel) at 1 V, you would get an average of 0.01 microamps. So the factor would reduce to roughly 10^5.

Wait a minute, a lot of those transistors are switching the same currents since they are in series. Also, FETs only draw most current while switching, so in between switches there's almost no flow of electrons. So in fact you cannot calculate things that way.

Yes, as I said the parallel assumption is not quite realistic, and the number is an average, covering all states a transistor may be in. So it amounts to a rough lower bound for when a transistor is switching.

Thank you for this - the name neural networks has made a whole generation of people forget that they have an endocrine system.

We know things like sleep, hunger, fear, and stress all impact how we think, yet people want to still build this mental model that synapses are just dot products that either reach an activation threshold or don't.

Fortunately for academics looking for a new start in industry, this widespread misunderstanding has made it only far too easy to transition from a slow-paced career in computational neuroscience to an overwhelmingly lucrative one in machine learning!

There have been people on HN arguing that the human brain is a biological LLM, because they can't think of any other way it could work, as if we evolved to generate the next token, instead of fitness as organisms in the real world. Where things like eating, sleeping, shelter, avoiding danger, social bonds, reproduction and child rearing are important. Things that require a body.

It's also frustrating because LLMs aren't even the only kind of AI/ML out there, they're just the kind currently getting investment and headlines.

I'm one of those people. To me those things only sounded like a different prompt. Priorities set for the llm

Isn’t that taken the analogy too literally? You’re saying nature is promoting humans to generate the next token to be outputted? What about all the other organisms that don’t have language? How do you distinguish nature prompts from nature training datasets? What makes you think nature is tokenized? What makes you think language generation is fundamental to biology?

Here's the hubris of thinking that way:

I would imagine the baseline assumption of your thinking is that things like sleep and emotions are a 'bug' in terms of cognition (or at the very least, 'prompts' that are optional).

Said differently, the assumption is that with the right engineer, you could reach human-parity cognition with a model that doesn't sleep or feel emotions (after all what's the point of an LLM if it gets tired and doesn't want to answer your questions sometimes? Or even worse knowingly deceives you because it is mad at you or prejudiced against you).

The problem with that assumption is that as far as we can tell, every being with even the slightest amount of cognition sleeps in some form and has something akin to emotional states. As far as we can prove, sleep and emotions are necessary preconditions to cognition.

A worldview where the 'good' parts of the brain (reasoning and logic) are replicated in LLM but the 'bad' parts (sleep, hunger, emotions, etc.) are not is likely an incomplete model.

Do airplanes need sleep because they fly like birds who also require sleep?

Ah a very fun 'snippy' question that just proves my point further. Thank you.

No airplanes do not sleep. That's part of why their flying is fundamentally different than birds'.

You'll likely also notice that birds flap their wings while planes use jet engines and fixed wings.

My entire point is that it is foolish to imagine airplanes as mechanical birds, since they are in fact completely different and require their own mental models to understand.

This is analogous to LLMs. They do something completely different than what our brains do and require their own mental models in order to understand them completely.

I'm reluctant to ask, but how do ornithopters fit into a sleep paradigm?

Great follow up!

Ornithopters are designed by humans who sleep - the complex computers needed to make them work replicate things humans told them to do, right?

It is a very incomplete model of an ornithopter to not include the human.

Here, it's actually fun to respond to your comment in another way, so let's try this out:

Yes, sleep is in fact a prerequisite to planes flying. We have very strict laws about it actually. Most planes are only able to fly because a human (who does sleep) is piloting it.

The drones and other vehicles that can fly without pilots were still programmed by a person (who also needed sleep) FWIW.

They do need scheduled maintenance.

Birds flap their wings and maneuver differently. They don't fly the same way.

People will spout off about how machine learning is based on the brain while having no idea how the brain works.

It is based on the brain, but only in the loosest possible terms; ML is a cargo cult of biology. It's kind of surprising that it works at all.

It works because well, its actually pretty primitive at its core. Whole learning process is actually pretty brutal. Doing millions of interations w/ random (and semi-random) adjustments.

I think I've fallen into the "it's just a very fancy kind of lossy compression" camp.

Honestly once you understand maximum-likelihood estimation, empirical risk minimization, automatic differentiation, and stochastic gradient descent, it's not that much of a surprise it works.

Ignoring for the moment that transistors and synapses are very different in their function, the current in a CPU transistor is in the milliampere range...

That seems implausible. Apple's M2 has 20 billion transistors and draws 15 watts at full power [1]. Even assuming that 90% of those transistors are for cache and not logic, that would still be 2 billion logic transistors * 1 milliampere = 2 million amperes at full power. That would imply a voltage of 7.5 microvolts, which is far too low for silicon transistors.

[1] https://www.anandtech.com/show/17431/apple-announces-m2-soc-...

A single precision flop is in the order of pJ. [1] A transistor would be much less.

[1] https://arxiv.org/pdf/1809.09206.pdf

How do you get to 10^10? I might be missing a fundamental of physics here (asking genuinely).

The more of our logic we can implement with addition, the more can be offloaded to noisy analog systems with approximate computing. It would be funny if model temperature stopped being metaphorical.

Synapses are very analog, but then the neuronal soma (cell body) and axon are pretty pulsatile again. A spike is fired or it isn't!

A simple discretization of the various levels of signal at each input/output, a discretization to handle time-of-propagation (which is almost surely part of the computation just because it _can be_ and nature probably hijacks all mechanisms), and a further discretization to handle the various serum levels in the brain, which are either inputs, outputs, or probably both.

Just add a factor 2^D transistors for each original "brain transistor" and re-run your hardware. Hope field effects don't count, and cross your fingers that neurons are idempotent!

Easy! /s

Modelling an analog system in digital will always have a combinatorial curse of dimensionality. Modelling a biological system is so insanely complex I can't even begin to think about it.

No neurons are the equivalent of transistors. Synapses are the equivalent of connections between transistors. The total neurons are about 100 Billion but the connections are 100 Trillion. A neuron can have up to 10,000 synapses, while a transistor can have on average only about 3. Impressive nonetheless on power efficiency.

So the actual equation should be 100 billion times 100 Trillion times clock speed? ( I have zero ideas about how our brains works )

Even we have 10 times higher clock speed, a trillion transistor GPU is still very slow comparatively speaking ?

Surely memory has to come in somewhere?

# synapses should be analogous to # model parameters no? And # model parameters should be linear in # transistors.

We can not even get close to saying our current networks can be even close to synapses in performance or functional because architecturally we still use feedforward networks no recursion, no timing elements, very static connections. Transitors will definitely have some advantages in terms of being able to synchronize information and steps to an infinitely better degree than biological neurons, but as long as we stick with transformers it's the equivalent of trying to get to space by stacking sand, could you get there eventually? Yes, but there's better ways.

>> # synapses should be analogous to # model parameters no?

I think they're equivalent to a parameter AND the multiplier. Or in analog terms they'd just be a resistor whose value can be changed. Digital stuff is not a good fit for this.

> Or in analog terms they'd just be a resistor whose value can be changed

For what it's worth, that's actually a thing (ReRAM/memristors), but I think it got put on the back burner because it requires novel materials and nobody figured out how to cost-effectively scale up the fabrication versus scaling up flash memory. I saw some mention recently that advances in perovskite materials (a big deal lately due to potential solar applications) might revive the concept.

> The power efficiency difference may be explainable by the much slower frequency of brain computation (200 Hz vs. 2GHz).

Partly, but also because the brain has an asynchronous data-flow design, while the GPU is synchronous, and as you say clocked at a very high frequency.

In a clocked design the clock signal needs to be routed to every element on the chip which requires a lot of power, the more so the higher the frequency is. It's a bit like the amount of energy used doing "battle ropes" at the gym. The heavier the ropes (cf more gates the clock is connected to), the more power it takes to move them, and the faster you want to move them (cf faster clock frequency) the more power it takes.

In a data-flow design, like the brain, there is no clock. Each neuron fires, or not, independent of what other neurons are doing, based on their own individual inputs. If the inputs are changing (i.e. receiving signal spikes from attached neurons), then at some threshold of spike accumulation the neuron will fire (expending energy). If the inputs are not changing, or at a level below threshold, then the neuron will not fire.

To consider the difference, imagine our visual cortex if we're looking at a seagull flying across a blue sky. The seagull represents a tiny part of the visual field, and is the only part that is moving/changing, so there are only a few neurons who's inputs are changing and which themselves will therefore fire and expend energy. The blue sky comprising the rest of the visual field is not changing and we therefore don't expend any energy reprocessing it over and over.

In contrast, if you fed a video (frame by frame) of that same visual scene into a CNN being processed on a GPU, then it does not distinguish between what is changing or not, so 95% of the energy processing each frame will be wasted, and this will be repeated frame by frame as long as we're looking at that scene!

> In a clocked design the clock signal needs to be routed to every element on the chip which requires a lot of power, the more so the higher the frequency is.

Clock only needs to be distributed to sequential components like flip flops or SRAMs. The number of clock distribution wire-millimeters in typical chip is dwarfed by the number of data wire-millimeters, and if a neural network is well trained and quantized activations should be random, so number of transitions per clock should be 0.5 (as opposed to 1 for clock wires), meaning that power can't be dominated by clock. The flops that prevent clock skew are a small % of area, so I don't think those can tip the scales either. On the other hand, in asynchronous digital logic you need to have valid bit calculation on every single piece of logic, which seems like a pretty huge overhead to me.

There's obvious potential savings in not wasting FLOPs recalculating things unnecessarily, but I'm not sure how much of that could be realized by just building a data-flow digital GPU. The only attempt at a data-flow digital processor I'm aware of was AMULET (by ARM designer Steve Furber), which was not very successful.

There's more promise in analog chip designs, such as here:

https://spectrum.ieee.org/low-power-ai-spiking-neural-net

Or otherwise smarter architectures (software only or S/W+H/W) that design out the unnecessary calculations.

It's interesting to note how extraordinarily wasteful transformer-based LLMs are too. The transformer was designed part inspired by linguistics and part based on the parallel hardware (GPU's etc) available to run it on. Language mostly has only local sentence structure dependencies, yet transformer's self-attention mechanism has every word in a sentence paying attention to every other word (to some learned degree)! Turns out it's better to be dumb and fast than smart, although I expect future architectures will be much more efficient.

IMHO the breakthrough will come with an analog computer.

The current systems simulate stuff through computation using switches.

It’s like real sand vs sand simulator in the browser. One spins your fans and drains your battery when showing you 1000 particles acting like sand, the other just obeys laws of physics locally per particle and can do millions of particles much more accurately and with very slight increase in temperature.

Of course, analog computations are much less controllable but in this case that’s not a deal breaker.

On the power usage, the difference is that those synapses are almost always in stand-by. The equivalent would be a CMOS circuit with a clock of minutes.

On the complexity, AFAIK a synapse is way more complex than a transistor. Larger too, if you include its share of the neuron's volume. And yes, the count difference is due to the 3D packing.

The brain’s 3d packing is via folding. Maybe something similar would be better than just stacking.

We also lose a lot when building computers due to the fact we have to convert the analog world into digital representations. A neural analog computer would be more efficient I think, and due to the non-deterministic nature of AI would probably suit the task as well.

Non-deterministic means random. AI or natural I is not random. Analog suffers immensely from noise and it is the reason the brain has such a large number of neurons, part to deal with noise and part to deal with losing some neurons along the way.

Non-deterministic doesn't mean random. Random means random. Non-deterministic means that specific inputs don't generate the same outputs. It says nothing about the distribution of the output values. A chaotic system isn't random, its just non-deterministic.

“Nondeternimism” gets used jn a variety of ways both from differing context and conflicting uses in the same context, but chaotic systems are fully deterministic in the most common relevant sense, but highly sensitive to inputs, resulting in even very small uncertainty in inputs to render them largely unpredictable.

Analog suffers from noise? What do you think you get when you convert analog to digital? Way more loss of signal than whatever inbuilt noise the analog signal had in the first place.

There are nueromorphic companies like https://rain.ai

interesting point.

i know of at least one startup working with that concept[1].

Im sure there are others.

1 - https://www.extropic.ai/

It has some interesting implications: If we manage to replicate a human-like brain in silicon, it could think multiple million times faster than the biological equivalent. So even if it needs a lot of energy to run, it could "out-think" us by orders of magnitude and would have crazy reaction times. Just imagine a sentient being with the human mental equivalent of millions of years in one year. Crazy.

But we know very little about how biological brains actually work and very few connectomes have been fully mapped as of yet. We still cannot fully explain how C. elegans with 302 neurons "thinks".

Also every neuron contains the entire genome, which is hundreds of millions to billions of base pairs depending on the animal and we also don't really know how important it is to the function of the brain.

So the "state" of biological brains is potentially utterly gigantic even for simple animals.

By some accounts it takes a pretty sizable neural network to simulate a single neuron:

https://www.sciencedirect.com/science/article/pii/S089662732...

So we are going to need a lot of computational power to approximate what’s going on in an entire human brain.

Apples to oranges. Gate count indicates nothing when the architectures are nothing alike.

Brain is a spiking network with mutable connectivity, mostly asynchronous. Only the active path is spending energy at a single moment in time, and "compute" is tightly coupled with memory to the point of being indistinguishable. No need to move data anywhere.

In contrast, GPUs/TPUs are clocked and run fully connected networks, they have to iterate over humongous data arrays every time. Memory is decoupled from compute due to the semiconductor process differences between the two. As a result, they waste a huge amount of energy just moving data back and forth.

Fundamental advancements in SNNs are also required, it's not just about the transistors.

>> 100 trillion synapses (very roughly analogous to transistors?)

Not even remotely comparable

* its unlikely synapses are binary. Candidly they probably serve more than one purpose.

* transistor count is a bad proxy for other reasons. A pipeline to do floats are not going to be useful for fetch from memory. "Where" the density lies is important.

* Power: On this front transistors are a joke.

* The brain is clockless, and analog... frequency is an interesting metric

Binary systems are going to be bad at simulating complex processes. LLM's are a simulation of intelligence, like forecasting is a simulation of weather. Lorenz shows us why simulation of weather has limits, there isnt some magical math that will change those rules for ML to make the leap to "AGI"

Transistor power is really not a joke. Synapses would take far far FAR more power at close to similar frequencies. Biological neurons are incredibly inefficient.

Biology is incredibly efficient at what it does well though. Thus only 20 watts of brain energy to coordinate everything we do. We didn't evolve to be mentats.

Neurons and synapses have more than 100 different neurotransmitters and their many receptors, there is reuptake and destructive encyme activity and they are connected up to many thousands of their peer. Every single neuron is a dizzyingly complex machine, employing countless sub machines and processes.

You can not reasonably compare this to model parameters or transistors at all.

Nature was “building” this stuff for eons. I feel pretty good about our progress in still less than a 100 years.

Neurons are not just linear algebra pieces by the way. This is why there is bibliography that talks about the complexity of a single neuron. So, comparing at the unit level at this point is apples vs. oranges.

But, yes, the brain continues to be a surprising machine and ML accomplishements are amazing for that machine.

I think biggest difference is millions of years evolutionary development. That's a lot of time difference.

Hmm maybe the frequency metric is off. If neurons can derive output from potentially N overlapping / multiplexed inputs .. it's as if the clock was N times higher too.

> is it just that we haven't cracked 3-d stacking of integrated circuits yet?

Yes. If we could stack transistors in the Z dimension as closely as we do in X and Y, we'd easily exceed the brains density.

I wonder if this trajectory will only lead to reinventing the biological brain. It is hard to imagine the emergence of consciousness, as we know it, on a fundamentally deterministic system.

> The power efficiency difference may be explainable by the much slower frequency of brain computation (200 Hz vs. 2GHz).

Or by a more static design. A GPU can't do a thing without all the weights and shaders. There are benefits of this, you can easily swap one model for another. Human mind from the other hand is not reprogrammable. It can learn new tricks, but you cannot extract a firmware from one person and to upload it to another person.

Just imagine if every logical neuron of AI was a real thing, with physical connections to other neurons as inputs. No more need to have a high throughput memory, no more need to have compute units with gigaherz frequency.

Why are static 3D cells going to get us there when other ideas have not? Is it needed to replicate “arbitrary” academic ideas of consciousness (despite our best efforts our models are always approximation) to make a useful machine?

“Living things” are not static designs off the drafters table. They’ll never intelligent from their own curiosity, but from ours and the rules we embed. No matter how hard we push puerile hallucinations embedded by Star Trek. It’s still a computer and human agency does not have to bend to it.