Hacker News Reader

Comment by tudorw | Hacker News Reader

If I understand this correctly, they added a property to a NN of distance, and set the training to penalize increased distance between nodes to simulate a physical constraint, under these conditions 'hubs' emerged which facilitated connections across distance, the other observation was that "the same node might be able to encode multiple locations of a maze, rather than needing specialised nodes for encoding specific locations".

The work suggests that existing approaches to neural network architecture would benefit from more closely emulating the operation of the brain in this regard.

View on HN

> "the same node might be able to encode multiple locations of a maze, rather than needing specialised nodes for encoding specific locations"

That may sound surprising, but it's actually not unusual. It's worth remembering that the difference between highly local encoding and everything encoding everything at the same time, is often a matter of a reversible transformation.

For example, take FFT of an image, and the result will be an image where each pixel encodes information about all pixels of the original image simultaneously. And Fourier transform - a shift from a time/spatial domain to frequency domain - is quite simple, very useful, and occurs in nature.

Sounds a bit like it's tending towards the behaviour of grid cells. Rather than encoding a specific, unique place, grid cells light up periodically in the latent space they encode, with different grid cells having different periods and different latent spaces. A given combination of grid cells uniquely identifies a thing/space/concept/location/datapoint/... token?

Of course for any given location this wraps around with a period of the least common multiple between the grid cells' periods (I think? Had to ask Bing about that one, he was helpful for once). This is why if you try to map locations you get a hypertorus.

FFTs can be useful when constructing CNN pipelines in unsupervised domain adaptation too. Fourier domain adaptation (FDA) uses the calculated amplitude of a target set to modify the source (train) set. Cool and quick to implement.

I wonder how pleased Joseph Fourier would be seeing how his work has become so pivotal in a world probably unimaginable to his time. Not just for ego sake, but just how damn useful something one's effort did becomes to everyone else.

I don't think it has to be the fourier transform necessarily.

I think the fourier transform might be a really approachable version because it is already in so much signal processing. Sometimes its periodicity can work against you.

From what I can tell you, what you need are:

1. invertible, identifiable basis

2. Something like Parseval's Identity

3. Able to "smooth" out things like the heaviside function or dirac deltas

4. Workable on graphs as well

5. decays at infinity

6. eigenfunctions are all orthonormal

Turns out, making something like that gets you something similar to the fourier transform, and the math is probably the simplest since you are working with easy exponentials that work nicely with convolution.

People have done similar work with Chebyshev polynomials, and there are actually a lot of applications in ML/AI using Chebyshev polynomials in graphical neural networks and triangulation.

edit:

The fourier transform also comes about pretty nicely in quantum mechanics because of poisson bracket stuff and its nice derivative properties for position-momentum space. I am don't think other transforms/function basis come out so naturally.

This is a great example why maths people can't carry on conversations with non-maths people. I'm sure everything you just said is very possibly true, but I have no idea. This would be like me asking ChatGPT a question, and it all sounds believable, but I have no way of checking if it is true.

Why do you expect to understand an expert’s talk, without yourself being knowledgeable about the topic/subject? With all due respect, this cockyness is unfortunately quite common with software developers (myself included), as if being good at one thing automatically translates to a different domain.

Nobel disease, but even without the Nobel-price?

Have you got some references on the Chebyshev applications?

https://github.com/mdeff/cnn_graph/issues/35

Gcns

Actually you dont need 5. just boundary conditions or decay.

Furthermore, the relationship between the representations in the two domains is governed by the uncertainty principle.

AFAIK, it's the opposite -- the Fourier Transform implies the uncertainty principle.

>is quite simple, very useful, and occurs in nature. I could not agree more, but what do you mean happens in nature?

Harmonic oscillators naturally sample Fourier transforms, so for example the different cilia lengths in our ears are used to transform sounds into the frequency domain and the amplitude of various frequencies is how our nervous system intakes sound.

> The work suggests that existing approaches to neural network architecture would benefit from more closely emulating the operation of the brain in this regard.

Does it? My read is that biological neutral architecture is a consequence of biological constraints that artificial NNs mostly don't face, so we don't necessarily need to try to copy it.

There's a reason airplanes were invented before helicopters.

Depends on your goal. If you want to simulate human intelligence, maybe you want to replicate constraints.

A fair chunk of AI work boils down to “make something that acts like a human.” On the other end of the spectrum is stuff that is more specialized, like very targeted classifiers; there is no reason to expect those would benefit from this.

Eh, maybe. The question there becomes, how much of human behavior is specifically a consequence of the physical constraints of laying out our neurons? I'm guessing not much. The article doesn't mention any change in output prompted by the change in internal architecture, which doesn't really confirm my guess, but doesn't falsify it either.

there's been physics simulation of bipedal models learning to walk via reinforcement learning. the results always were a bit choppy and robotic, until they implemented signals propagation delays. this led to more natural, fluid movements that definitely looked more "human". sorry, can't find the video.

personally i absolutely do think that for generating convincingly human-like intelligence you also need some human constraints, otherwise you will get some uncanny valley.

another example would be alpha-zeros play style. AIs don't play like humans, they maximize their chance of winning in the long term without going for good looking opportunities that hurt their chances in the long run (like human players do).

Did the bipedal walking people try optimizing for energy instead? If so, and the models were still choppy, maybe that's actually better.

I guess I just find the goal of imitating human intelligence including all its mistakes to be a silly goal. The only time you want that instead of an actual human is if you're trying to deceive people into thinking your AI is a human. Otherwise, you just want the correct answer (or, if you're afraid, you want a strictly sub-human intelligence).

Has there been research on whether it's possible to similate human intelligence without also getting all of its flaws? Susceptibility to logical fallacies, random difficulty with factual recall, fading memory over time, biases, prejudice, etc.

That's something you could philosophise about but not research unless you already have a human level intelligence to test. We won't know if it's even possible to replicate in silicon for a very long time

No, AI work boils down to “output computation results that humans hallucinate could have been generated by a human””

We’re just normalizing to our innate sensibilities. We have no idea if we’re making intelligence or what that even means as us humans must work within the constraints we evolved into. We have no idea if we generalized consciousness as it could exist across space time.

You and I will never exist outside our universe and observe what makes it tick. We’re hanging out on Earth making mannequins talk, hallucinating we’re gods because of it. Humans are artificial intelligence given their lack of direct observation of so much of the universe.

Conway came up with his game of life after the universe. Sorry, AI researchers, life, consciousness and visualization were already created by reality. We’re just working on an easy to use Dewey Decimal system to catalog it.

We don't need to copy but I think the hypothesis is that constraints like ours help networks optimize towards outcomes like ours, and it seems to test well in this case.

That's true, but it's a quite distinct hypothesis from the bit I quoted.

You're dead on.

We've developed numerous different learning algorithms that are biologically plausible, but they all kinda work like backpropagation but worse, so we stuck with backpropagation. We've made more complicated neurons that better resemble biological neurons, but it is faster and works better if you just add extra simple neurons, so we do that instead. Spiking neural networks have connection patterns more similar to what you see in the brain, but they learn slower and are tougher to work with than regular layered neural networks, so we use layered neural networks instead.

Biology is Biology and Silicon is Silicon. Sometimes, Constraints can be just that - Constraints and not some secret sauce.

I'm interested in the 'slower/tougher' route you mentioned, do you recall the final performance, am assuming it was equivalent or worse. Given the vast disparity in power consumption between silicon and meat space it's surely a rich area to explore, especially given the feedback loop present where AI is being effectively used in more accurate observation and interpretation of brain activity.

Also, biological “architecture” is so different compared to silicon design, that it might just the case that different algorithms perform best for the respective design. Biological systems are effectively huge distributed computing systems, where each unit is very weak. Backpropagation probably can’t easily be implemented for that.

I agree with this, I should have been more specific, I really meant it from the researchers perspective, they are trying to make NN's that more closely model the brain for their brain research, not for AGI, so in that regard, the research was looking at how constraints affect learning, and whether applying constraints might result in a network that more closely mirrors brain activity, for AI purposes, we want smart machines, not necessarily a model of the brain with all it's additional functions aside from 'intelligence' :)

If this is the case, the finding sounds trivial. I thought it was literally a theorem that hubs emerge from networks if you minimize distance or wiring cost.

Maybe, I think the research here is from a brain perspective, AI and Brain Science are having an interesting time, with learnings from both feeding in, so this article is focused on the physicality of networks in the brain, and looking at whether by increasing the alignment between the way a neural network works and the way brains work they can develop a 'digital brain', for me this is distinct from machine intelligence, this is emulating brain activity to provide cognitive and psychological insight in meat space.

Topology is hard for me, visualising multi-dimensional topological manifolds even more so, however, I am intrigued by the opportunities, we use graphs and other forms of visualisation to reveal topology, and it would seem reasonable that behaviours of NN that are 'transferrable' e.g.. trained on one set of data and able to operate on another category of data, may be a 'shape', a complicated one, but perhaps one that would once revealed would aid in the understanding of what happens under the hood.

There must be some difficulty in studying biological brain intelligence to distinguish between features that contribute to intelligence and those which are simply path dependencies from millions of years of nervous system evolution.

Over that time, brain function was mostly concerned with aiding the organism to find food, grow, reproduce, and avoid being eaten, rather than language, logic, mathematics, arts, and so forth. Its rather astonishing that humans are somehow now able to do the latter, using brains evolved to do the former.

Why would there be a difference at all here? Finding food and avoiding being eaten requires a great deal of intelligence, is surely the basis of human logic, and is the foundation for language. It’s all one interconnect network and every bit of it is a piece of the greater intelligence.

It isn't just humans that can do math. Ants to geolocation [0] by counting their steps in relation to the distance they walked from their nest. They do this with just a small amount of neurons.

[0] https://blog.cambridgecoaching.com/ants-go-marching-fun-fact....

Chickens can count. Humans can do abstract math, any math.

Well, humans can do any math that human brains can comprehend. That doesn't mean "any math" any more than if a chicken we're to claim that chickens can do any math (because they cannot comprehend human math).

You're overstating human difference.

We appear to have a capacity for substantially greater sophsitication in those domains, but none are unique to us except when we artificially define them to be. Remember that words like "language", "logic", "art", etc are cultural inventions with a fuzzy and fluid relationship to whatever real-word "stuff" they refer to, not natural kinds that themselves have sharp and perennial definitions.

Unless you choose to define the word as that which only humans can acheive, a spider's web elegently reflects "mathematics" just as much as some beautiful proof in set theory; a conflicted bird debating itself over which stem to use in its nest reflects artistic attention just as as a painter choosing their next color; a cat chirping or mewing or yowling reflects language just as me writing this comment.

The sophistication doesn't go as far, by our eye at least, in any of these animal examples, and so we don't expect the spider to confirm Fermat's Last Theorem or the bird to feature their nest in a gallery (actually...) or a cat to compose formal poetry, but the essential bits that we extend with our sophistication are all ancient and widespread throughout nature.

It's still astonishing that any life can do so many of the things it but I guess that's apparently what billions of years of "pretraining" on unfathomably efficient machines gets you.

Incidentally, it's wild to see people believe that a stream of fmults pushing through a trillion transistors would get you even close to the sophistication of any of life's intelligence. For current-AI-skeptical materialists, it's usually not a doubt about whether silicon and software might conceivably be intelligent, but it can just seem absurd to believe the grossly crude and narrow innovations of recent years are even close. You need to have a very shallow, narrow, almost willfully blinded, appreciation of the "intelligence" exhibited throughout all biological life to think that you unlocked the silicon version of it all in a pretty-good chatbot running on Azure.

This is a little too hard core anthropomorphism for me.

no need for billions of years of pre-training, they were designed that way from the start :)

Full of bugs?

I think the the emergent behavior is still the most plausible. There really is not much difference between our brain and a chimpanzees, it is definitely not accounted for genetically to be that much more intelligent.

> existing approaches to neural network architecture would benefit from more closely emulating the operation of the brain in this regard.

From https://news.ycombinator.com/item?id=38334538#38336861 :

> Which NN architectures could be sufficient to simulate the entire human brain with spreading activation in 11 dimensions?

I'm just surprised this hasn't been done before. Isn't this the kind of low-hanging fruit that would have been studied at some point in the last 40 years?

It was ... look up "AI winter". Vastly simplified, we collectively ignored ANNs until somebody figured out that we now have enough recourses to create larger ANNs than before.

Definitely familiar with the AI winter...which is why it seemed odd this current research as being presented as new. Guess it's just a rehash of the old ideas.

Which is a fine place to start, a lot of our current shiny toys are a rehash of old ideas in a modern context, https://people.idsia.ch/~juergen/deep-learning-history.html

You should post that here. (if it is not something that is already posted again and again).

Reply on news.ycombinator.com