Multimodal Neurons in Artificial Neural Networks
openai.com
openai.com
I guess we see something like this:
e.g. "Photo of spider" -> Hierarchy of pixel soups -> "Photo of spider" OR "Photo of word spider" OR "Spider rear view" OR "Spiderman" OR ... -> [Spider]
"Photo of spider" -> "Characteristics of a spider" -> [Spider]
"Photo of word spider" -> "Letters S-P-I-D-E-R" -> "Written word spider" -> [Spider]One is just basic generalisation - do these neurons effectively capture things within their semantic group (whatever that means) but completely outside of the training data. If yes, then I guess the answer might be 'yes, in some sense it is like a "real abstraction"'.
Second, and (afaik) currently a more philosophical framing - it isn't obvious whether "(sufficiently advanced) OR logic" and "real abstraction", are actually different. Additionally, for the purposes of a model like this one, I find it hard to see how they could be different. The best the model can do is (roughly speaking) assign neurons to particular concepts, be they ones that fit with our mental models of the world, or ones that are more "functional". The better a job it can do of the former, the more we might be inclined to believe that it is modelling things as we understand them.
When I talked about OR logic I meant a simple and direct aggregation of all possible pixel soups that should be labeled in a specific way. Calling this multimodal and comparing it to the Jennifer Aniston neuron is framing the situation in a way that is not good for science (see AI winter). Especially when multimodal neurons in neuroscience refer to processing more than one sense.
I find it hard to imagine a different solution than sufficiently advanced OR logic for real abstraction within the current ANN models. That does not mean there is none within ANNs and especially not in the brain. We are far from being able to do anything like the brain does and don't know how it learns new concepts so fast - without thousands of sample images. Just because we observe an effect that reminds us of some vague findings in the brain and we call the effect the same in ANNs does not make ANNs more like brains.
BTW nice podcast that also discusses the Jennifer Aniston neuron with its "discoverer": https://soundcloud.com/neuropodcast/episode62
Seeing the Spider-Man neuron work on multiple types (pictures, drawings, text), makes it seem like we can teach AI to learn these same type connections.
And if we scale up the network size enough, what if we could see these types through the equivalent of a being with 1000IQ? What connection types are the most effective for a being like that? Can we even understand them? Maybe they would be deep, and archetypical in the way that Odysseus and Harry Potter are the same, despite the fact that one is an ancient Greek king, and the other is a modern British wizard. Even more interestingly, maybe the connections would be completely inexplicable to us, with no apparent rhyme or reason perceptible to humans.
One of the amazing things about this project exploring CLIP was seeing some hints of this. For example, one day I was studying one of the Africa neurons and it generated the text "IMBEWU" -- it turns out this is a popular TV show in South Africa (https://en.wikipedia.org/wiki/Imbewu:_The_Seed). That's a trivial example, but it begins to hint at something interesting.
I'd really love to see what a domain expert analyzing CLIP would make of things. For example, I'd love to hear what ethnographers think of the region neurons, or what historians think of the time period neurons. Especially for future, larger models.
That would be analogous to: What kind of tokens when put together make state of the art algorithms? - they are just tokens, even if it's the best implementation.
Intelligence is in the game, including the agent, the environment and goals. It's not in some kind of special neuron connections, it's in the way the brain is connected to opportunities and dangers outside.
In theory, it appears that the articles are supposed to be self contained Git repositories. In practice, this article loads a bunch of assets from Google's Cloud Storage. Blocking them appears to partially break various bits in subtle ways - figure 6 is particularly bad. It also fails to render correctly without a couple third party scripts hosted by Cloudflare. It's hardly an ideal state of affairs.
Articles that aren't self contained make it frustratingly difficult to view offline copies. They also raise the possibility of the publication changing over time as anything hosted by a third party is outside the control of the journal. This lack of an immutable version is highly concerning because it means that no single canonical form exists for incorporation into the scholarly record. (Note that even redacted, entirely fraudulent papers have traditionally remained available from the original publisher with appropriate warning labels attached.)
Reflowable text and interactive figures are very welcome additions, but the concept of a mutable scientific literature is absolutely horrifying.
I have been unable to get this to work. Does it work for anyone else?
(Each neuron has a dropdown menu, but it sounds like there's more to it.)
[1] https://microscope-azure-edge.openai.com/models/contrastive_...
[2] https://microscope-azure-edge.openai.com/models/contrastive_...
The finance neuron [1330], for example, responds to images of piggy banks, but also responds to the string “$$$”. By forcing the finance neuron to fire, we can fool our model into classifying a dog as a piggy bank.
"Our model, despite being trained on a curated subset of the internet, still inherits its many unchecked biases and associations."
If these models find themselves into production environment - if they are good enough and profitable enough - they will eventually become legacy systems quietly perpetuating the biases of past times.
So these models for now can be treated as a very reductive snapshot of humanity (at least English-speaking segment), warts and all
The model learning bias isn't the issue. You could ask me what I think the racist members of my family might write about a given image. I'd then be able to emulate them inside my head and accurately predict their responses. We all do that. It's how we have moments like "I knew you were going to say that."; "That's typical of you to say."; "Why am I not surprised?"; etc. The fact that I, and everyone else, can do that does not imply that we are biased. It's how we behave that determines if we are biased.
We want our AIs to do the same.
The real ethics question here is not how to we prevent AIs from learning bias. It's how do we get AIs to not _express_ those biases. We need a way to put them into "impartial" mode, much like we take biased and fallible humans and make them judges in courtrooms.
Personally I don't think that's going to be as hard as some imagine. Again, remember that these AIs are learning to emulate humans, _including_ judges. Give GPT-* a bunch of court documents and transcripts and it will learn the capacity to emulate a judge. Then you just need to carefully craft its prompt text so that for any given query, you can be reasonably sure it's acting impartially.
The irony is that the black bias mentioned in this paper is probably due to inherent bias in image processing algorithms of the sensors themselves. Look up the "Shirley Card"
As for prompt engineering for gpt, it feels a bit like reading tea leaves. I'm not sure if it is possible to know for certain that a specific prompt will elicit the desire all the time.
They say "Each neuron is represented by a feature visualization with a human-chosen concept labels to help quickly provide a sense of each neuron", and these neurons are selected from the final layer. I don't think I understand this.
The ones you see in this work are mostly a variant of the standard feature visualization, which tries to show different "facets" in neurons that respond to multiple things. The details are explained in the appendix of the paper (https://distill.pub/2021/multimodal-neurons/ ).
Has there been any study of variability in these activation images - like are there many disconnected local maxima depending on the initialization, or how they vary with retraining the network (or e.g. with dropout, etc), or varying the model parameters in some direction that keeps the loss in a local minimum.
I could picture that maybe they always look the same, but sometimes there would be cases where they have different modes that accomplish the same thing.
In more recent research, people noticed that sensible labels can be given by seeing what sort of inputs activated it. For example, if that neuron lights up with any picture with a lot of right-angles, and nothing else, a researcher may call it “the right-angle neuron”.
In this case, the researchers did this to the CLIP network and found it was much easier than usual for humans to come up with sensible names for the type of thing any given neuron would recognise.
You can also demonstrate what a neuron does by running the maths in reverse to get an input which would activate that neuron more strongly than any other input; in this case an image.
Isn’t this also how neuroscientists identified brain regions?
I guess it makes sense this would happen but it’s curious that we’ve been able to build a blackbox algorithm/technique that’s as difficult to reverse engineer as real brains.
I'm not suggesting this example is the case, but with AlphaZero, for example many of the moves are beyond an average players comprehension. What happens when it's beyond the greatest of us?
Exciting times are ahead!
https://theconversation.com/singularity-how-governments-can-... provides some starting points.
I'd agree with that, but I wouldn't be surprised if some narrow domains are completely surpassed.
https://arxiv.org/abs/1705.08807, https://www.youtube.com/watch?v=HOJ1NVtlnyQ
AGI would involve having some motivation to stay alive, affect the world and imagine things, instead of just using a fixed amount of CPU to point out whether something looks like a cat.
From the article:
> Unlike humans, CLIP can’t slow down to compensate for the harder task. Instead of taking a longer amount of time for the incongruent stimuli, it has a very high error rate.
Anyhow, I suspect you're not familiar with ML research, since eg. there are plenty of non-feedforward networks available. I also don't think much of this really matters; it is absurdly anthropocentric to expect ‘motivation to stay alive’ to be fundamental to intelligence.
Good researchers will never rule it out of course but I don't know any who think it is likely (I work in the field).
I can give more but I did upstream link to a paper on this so it seems pretty pointless.
I can point at "Geoffrey Hinton and Demis Hassabis: AGI is nowhere close to being a reality"
https://venturebeat.com/2018/12/17/geoffrey-hinton-and-demis...
and
"Martin Ford interviewed 23 of the most prominent men and women who are working in AI today, including DeepMind CEO Demis Hassabis, Google AI Chief Jeff Dean, and Stanford AI director Fei-Fei Li. In an informal survey, Ford asked each of them to guess by which year there will be at least a 50 percent chance of AGI being built.... Ray Kurzweil, a futurist and director of engineering at Google, suggested that by 2029, there would be a 50 percent chance of AGI being built, and Rodney Brooks, roboticist and co-founder of iRobot, went for 2200. The rest of the guesses were scattered between these two extremes, with the average estimate being 2099 — 81 years from now."
https://www.theverge.com/2018/11/27/18114362/ai-artificial-g...
I don't think there's a single timestamp for this since the exact question was never asked. But the gist is very explicit in his numerous talks (which note includes both the Future of Life Institute and Singularity Summit), and I think https://youtu.be/d-bvsJWmqlc?t=203 sums it up beyond reasonable doubt.
> I can point at "Geoffrey Hinton and Demis Hassabis: AGI is nowhere close to being a reality"
This is a disaster of a headline, since while Hinton does, Hassabis never says that.
> The rest of the guesses were scattered between these two extremes, with the average estimate being 2099 — 81 years from now.
Which agrees with my point.
It's an aspiration, but not something anyone believes on the horizon.
That's not AGI.
Note that timelines for the HLMI variant of the question[1], which was asked to more people, had more like a 50% aggregate probability for 50 years away, per Figure 1.
[1] “High-level machine intelligence” (HLMI) is achieved when unaided machines can accomplish every task better and more cheaply than human workers.
It isn't surprising that a model trained for alligning photos and corresponding text labels has neurons that have learned higher semantic concepts as that is exactly the goal of generalisation.