Hacker News Reader

Comment by electric_mayhem | Hacker News Reader

It’s literally simulating a neural network.

How much of your 5-sense experiential memories and decades of academic book learning are you bringing to understand my reply to your post?

How many gigabytes do you think that’s equivalent to?

View on HN

Jamba seems to be distributed as 21 5-gigabyte files [1] so I guess that’s another way of looking at it.

[1] https://huggingface.co/ai21labs/Jamba-v0.1/tree/main

So what? I have seen models distributed as 26x 10GB files.

It’s kinda simulating our brains but not really. When I attempted to dig more into how neurons work I realised that it’s a massive chasm of difference. Very much worth doing if you haven’t (you might know far better then me, this is for people who don’t yet.)

In terms of results: Our brains are working with 20w of power and can be trained to compete with LLM’s using a tiny fraction of the world’s data. They also have to keep you breathing and your blood pumping and manage all the dangers of catching a ball near traffic. Or skiing, or poetry, or sunsets. And they remember stuff five minutes later and don’t need a training run that takes months.

We have SO many opportunities to improve the AI architecture it’s ridiculous. This is a good thing.

To be fair most of the brain is more like a pretrained model — it isn't being trained at any point after conception to keep your blood pumping or your lungs working, it does that out of the box roughly as soon as you sprout those organs (or the minute you're born, in the case of lungs). The training process was billions of years of evolution. And, well, given fairly persistent cross-cultural cognitive biases, I expect the conscious thought parts are starting from a pretrained model, too, and all we're doing in school is finetuning ;)

People don't understand that to simulate a single neuron, you need an entire neural network. So 70 billion parameters might at best be equivalent to a million neurons but that is assuming that your neural network architecture is akin to the connections between neurons. Considering the physical sparsity, you might need even more parameters to model the connections of a biological neural network. So less than a million neurons in practice.

I love both parent post perspectives on this.

Reply on news.ycombinator.com