Oh woah, thanks for catching that! The ability to change each neurons facet on hover was added towards the end, but seems to be blocking one from clicking the <a> tag.
1,213 karma · joined October 10, 2014
http://distill.pub http://colah.github.io
The ones you see in this work are mostly a variant of the standard feature visualization, which tries to show different "facets" in neurons that respond to multiple things. The details are explained in the appendix of the paper (https://distill.pub/2021/multimodal-neurons/ ).
One of the amazing things about this project exploring CLIP was seeing some hints of this. For example, one day I was studying one of the Africa neurons and it generated the text "IMBEWU" -- it turns out this is a popular TV show in South Africa (https://en.wikipedia.org/wiki/Imbewu:_The_Seed). That's a trivial example, but it begins to hint at something interesting.
I'd really love to see what a domain expert analyzing CLIP would make of things. For example, I'd love to hear what ethnographers think of the region neurons, or what historians think of the time period neurons. Especially for future, larger models.
Just to clarify, impact factor measures the average impact of a paper, not the total impact of the journal. In fact, it divides by the number of papers! So it's much easier to have a high impact factor if you're very small, and it's probably not quite the right way to think of whether society should invest resources in something.
(In fact, we technically don't have an IF, just the value that would be an IF if we were bigger. IF isn't officially defined until you reach a certain number of publications a year.)
> wouldn't moving towards more robust and professional editorial staff be beneficial for the large audience? Or do you think there's a benefit too having volunteers?
I think there's two main things.
Firstly, it's just a very difficult transition to make. If we wanted to hire staff, we'd need to find people who were aligned with Distill's vision and wanted to dedicate their carer to it. Hiring is hard! We'd also need to find a reliable funding sources (probably at least 200K/year) that was reliable (we don't want Distill to suddenly disappear!) and dosn't impose negative externalities (like being closed).
Secondly, it's a lot harder to experiment when you're big and have commitments to more people. It's already quite hard for Distill to do weird things now -- a lot of people get annoyed to us, and I feel a visceral pressure to be conservative.
Also, a lot of our ability to do weird things and have beautiful articles comes from expert attention to individual papers. A lot of early Distill papers involved Shan Carter or I spending 50-100 hours mentoring authors and giving them design and writing feedback. There's no model where that make sense at scale.
For regular Distill, all the costs (several thousand dollars a year) come out of volunteer pockets, and all the time is volunteered.
Edit: There was a period of 6 months where Firebase (which is connected to Google?) gave us free hosting, reducing our expenses.
I genuinely appreciate you giving feedback. Running Distill is hard, and there's lots of places we make mistakes or could do things better.
From my perspective, Distill has to balance a lot of equities. I see one of our major roles as trying to explore new visions of what an academic journal can be, so I put a lot of weight towards us doing weird and unusual things, like these threads. But we also need to take care around process and quality. And sometimes we screw up.
Distill is also entirely volunteer run. Everyone involved has full time jobs, and runs Distill on top of that. And honestly, we're often quite exhausted. So sometimes we make mistakes that a better version of ourselves with more time for Distill would not.
It seems like it's important to you to call Distill out on what you see as an institutional failure. I think you could actually do that more effectively and credibly if you wrote your comment differently.
Putting myself in the author's shoes, it feels pretty hard to engage with your comment. When you swear, it gives the sense you don't wish for people to engage with you. When you say "who the fuck reviewed this paper, a 25 year old fresh professor", it feels like you're implying that people who are young are intrinsically not knowledgable. When you don't leave open space for more charitable interpretations, it makes productive conversation a lot harder.
I appreciate your dedication to improving scientific discourse. I'd love to see if we can turn this into a productive conversation.
I understand the task to be different from regular pixel segmentation in that it's set up in the cellular automata context and message passing is only allowed between neighbors. I think this is what the authors intend by "each pixel in the digit needs to coordinate locally".
With that said, you bring up a good point that it could be misread. I bet the authors would be happy to update the paper and clarify if you wanted to engage!
Any chance you could consolidate this one into that thread?
Firstly, make sure you click through to the invidudal articles and try the interactive demos at the top of each one:
* https://distill.pub/2020/growing-ca/ * https://distill.pub/2020/selforg/mnist/
I really enjoy how quirky Alex, Ettore, et al's experiments are. It isn't going after benchmarks. It has a very "what would happen if we tried this?" feel to it. If I had to bet, I don't think it's super likely to have direct real world applications, but it's immensely fun to read about and expands the space of ML systems I might imagine.
I've also been enjoying the open slack channel associated with this thread. It's pretty low volume, but there's been a number of fun experiments from people who enjoyed the original article on neural cellular automata. I feel like there's often a lot of really cool experiments and conversations that no one outside a given lab sees, because they don't make it to publication, and I like how the channel allows those to be shared with anyone really excited about this line of work. It was also really heartwarming to see people jump in to give feedback on the draft of the most recent article!
Regarding DeepDream, it often feels to me -- I don't wish to speak on behalf of Alex or Mike -- that we didn't really understand what our results meant when we published DeepDream. It was kind of like discovering that warped glass can distort and magnify images: a really interesting discovery, but a lot more work was needed to turn it into a scientific instrument like glass can be used to form a microscope. As the community got single neuron or direction feature visualizations that worked well, lots of research possibilities began to open up. And in retrospect, one of the most important tricks was jitter, which Alex introduced. This style of feature visualization is probably the single tool I rely on most in my research to this day.
(If you're curious what this has led to as we've continued to pursue it, check out Circuits (https://distill.pub/2020/circuits/zoom-in/), Building Blocks (https://distill.pub/2018/building-blocks/) and Activation Atlases (https://distill.pub/2019/activation-atlas/).)
I'd also encourage people to check out Alex' new line of research, Neural Cellular Automata (https://distill.pub/2020/growing-ca/). I think it's a really interesting line of exploration. And as usual, Alex has an incredible deep trove of small fascinating results relating to NCA if you talk to him about it.
A notable researcher privately told me that they think all interpretability research is nonsense. As someone who's dedicated the last six years of my life to this field, that was pretty uncomfortable to hear. But I think it's important to pay attention to, because I think it's actually a pretty common, unspoken view.
As a result, this has been on my mind a great deal. I think two important questions are:
(1) How can we surface the disagreements that are leading to such divergent views between different members of the research community? (Especially when people are generally too polite to say that they think something is total nonsense.)
(2) What would a more epistemically stable foundation for interpretability look like?
I'm not sure what the right answers to these are, but I think they're important to discuss.
Penrose notation is very high level: edges correspond to tensor dimensions, joining corresponds roughly to tensor contraction. When you're working with multiple complicated multi-dimensional tensors, it can be a beautiful way to express things.
This notation seems to be expressing something a bit lower level: it seems that edges correspond to values and dots correspond to copy/add. So, roughly, they visually express linear functions. (I didn't read the full post, it's possible that they expand beyond this initial meaning.) Diagrams of a similar style are used in other contexts (eg. expressing the weights of neural networks), often as a pedagogical tool.
One thing we're thinking a lot about how you can transition from "what" a neural network represents to "how does it mechanistically do that"? I hope we'll publish on that in the next couple of months.
By the way, while I'm flattered by the esteem it implies, I'd generally rather people not refer to us as the "Olah Team", just as you wouldn't refer Google Brain as "Jeff Dean Research" or MILA as "Yoshua Bengio Institute" or OpenAI as "Sutskever AI", etc. I think academic culture of branding groups after the PI is kind of unhealthy.
My teammates are doing the hard work and I see my position as a team lead as just to serve and support them. :)
It feels a bit awkward to chime in, but it feels a bit important because I think there's a way in which the narrative is a little off (although I'm very grateful for everyone's kind comments):
Working at Google wasn't a trade off where I got financial support in exchange for not publishing. Rather, working at Google was amazing enabler, which supported me doing work I believed advanced important public interests for five years. That actually includes the blog posts being discussed here, which I wrote while an intern on Brain and received feedback from many researchers there.
To be concrete, some of the things I did while at Google included:
* Writing expository blog posts explaining important ideas in machine learning (eg. this blog post).
* A five year thread of research on neural network interpretability. How humans can understand how neural networks make decisions? (eg. DeepDream, Feature Vis, Lucid, Building Blocks)
* Cofoudning a scientific journal for machine learning that values exposition and allows interactive diagrams.
* Working on early TensorFlow and writing some of the initial tutorials on it.
* Articulating concrete safety concerns about modern machine learning systems, organizing a cross-instutional paper on it, and later representing Google on the Partnership on AI's Safety-Critical AI working group.
* Designing and teaching the introductory course for the Google Brain/AI Residency.
* Miscellaneous ML research, all published in open access venues.
That's basically the majority of what I did at Google. I always felt empowered to work on things that I thought were important for the world, and in many cases felt like had leverage I wouldn't have had individually. I think that was exceptionally generous of Google, and also speaks a lot to the environment Jeff created for those of us on Brain. (I realize Google is a large company, and other people's experience may be very different.)
I do think you're right that the number of short expository blog posts I wrote declined, especially after my first two years at Google. A big part of that is that I focused more on a small number of more ambitious projects (eg. Distill, Building Blocks). Another big part is that I started doing more non-individual work: teaching, mentoring, editing, debugging social issues.
To be clear, I actually left Google last fall. I now work at OpenAI where I lead the new Clarity team, which works neural network interpretability -- basically, can we take a trained neural network and turn it into something like code that a human can understand? Leading a team means that I do even less individual research and writing. Sometimes I miss being an individual contributor, but I think it's the right call. I get to build an environment where others can focus on a kind of research I think is really important for the world and teach and support them.
Unfortunately, the Fellowship only lasts for two years. Towards the end, it was actually super stressful. How could I support myself while continuing to work on what I thought was important? And if I wanted to live in the Bay Area, how could I event get immigration status to work and live there?
Getting to work at Google Brain at this point let me continue to do what I was doing before -- many of my blog posts, including the one being discussed here, were written while I was at Brain.
> This article was presented at VISxAI, it is now under review at distill.pub.
An earlier version of this paper was presented at the VisxAI workshop, and then later expanded into a journal paper published in Distill.
(Disclosure: I'm one of Distill's editors, but was not involved in editing this particular paper.)
As I understand it, a big part of the green revolution, especially initially, was Borlaug developing a strand of wheat that was resistant to many common diseases and had higher yields. This seems born out by a quick skim of his [wikipedia article](https://en.wikipedia.org/wiki/Norman_Borlaug), although I'm sure nitrogen fertilizer, etc, also played large roles.
I don't have the time to do a deep dive right now, so I've just updated my previous comment to be uncertain.
[1] See discussion below about relative importance of engineering disease resistant varieties vs fertilizers and pesticides. I only have a vague familiarity with the green revolution and could be mistaken.
https://blog.openai.com/deep-reinforcement-learning-from-hum...
I think it's likely that deep learning has stumbled upon something deep and profound. And now we're at the point of struggling to make sense of it. Of course things are messy: we're knee deep in the business of trying to start to sort things out.
In the optimistic view, the ideas we're grappling with -- ideas like feature visualization, attribution, etc -- might be the seeds of deep abstractions like calculus or information theory. (Of course, these early versions are messy! For example, early calculus was deeply criticized by figures like Berkeley, and took more than a century to put on firm footing via the introduction of limits.) Powerful, novel abstractions may not look like what you expect at first.
I do think it's very reasonable of you to be skeptical, of course. Most attempts to craft new ways of thinking about hard problems don't pan out. But I think it's worth pushing really hard on them, because if they do they're very valuable. I feel like we have initial promising results -- give us a decade to see where they go! :)
But we could also be totally barking up the wrong tree. :)
> What's more- this only works with vision, where activations can sort of map to images. It's no use for, say, text, sound, or other types of data (despite what the article says).
That's a reasonable concern. We didn't give any demonstrations of our methods outside vision in the article. I can say that we have done very early-stage prototypes that suggestion similar interfaces work in other domains. Of course, instead of images you get symbols in what ever domain you're working with -- such as audio or text.
As with many ideas in this article, Alex deserves all the credit. He's been doing that trick internally at Google for years. (In my experience, if Alex is excited about an idea, >50% odds you'll realize it's a super important idea 2 years later. :P )
I think there's I saw an instance of someone doing PCA on conv net activations to make heat maps. We should try to dig that up and cite it.
> I suspect this kind of whole-DNN matrix decomposition will be widely applicable across architectures and modalities, not just for convnets in visual tasks.
Absolutely! (I think that most of the ideas in our article are pretty general. :) )
> This is exciting stuff, and I can't wait to see what kinds of interpretation tools (and UIs) you and others come up with in the near future.
Thanks! We're super excited as well!
> Finally, I can't help but wonder if the behavior of state-of-the-art AI systems has already exceeded, or is on the verge of exceeding, human capacity to interpret it... What is your view or opinionated guess on this?
Oh my. That's a super interesting and deep question. Wild wild speculation ahead. Please take everything I say with a big grain of salt.
My first comment is that a model can be hard to understand because it's too stupid, in addition to being hard to understand because it's exceeding us. I suspect that if one could have an "interpretability curve" of how easy models are to understand vs model performance -- it's kind of hard to make this sensical, because both variables are actually these really nuance high-dimensional things, but imagine you could -- it would go up as you approach human performance and actually peak after it, before eventually declining again. This intuition is driven by the thought that early superhuman performance for tasks humans are already good at is probably largely about having really crisp, much more statistically precise, versions of our abstractions. Those crisp versions of our abstractions are probably easier to understand than the confused ones.
But, of course, that one dimensional view is a gross simplification. I suspect that something that will be important in future thought about this is "alien abstractions" vs "refined abstractions."
By a "refined abstraction" I mean something like ears in GoogLeNet. You see, GoogLeNet has dozens of detectors for different kinds of ear -- a much richer vocabulary of ear types than I can articulate -- and knows a lot about how those should influence class probabilities. Although I can't understand the detailed nuance of each ear detector, I can get the general gist and verify that it has reasonable consequences. Conversely, by an "alien abstraction" I mean a feature that I don't have a corresponding idea for. These are much harder for us to deal with.
Both "refined" and "alien" abstractions could give superhuman performance. We're in a much better state when refined ones dominate. For visual tasks that humans are already good at, I expect refined abstractions to dominate for a while. In other domains, I have a lot less confidence.
I think there's a lot of nuance about where interpretability will easily scale vs have severe challenges is super subtle. I have a draft essay floating around on the topic. Hopefully I'll get it out there someday.