Could a Neuroscientist Understand a Microprocessor? (2017)
journals.plos.org
journals.plos.org
That the tools of neuroscience choke on a 6502 is as much of an indictment of the former as my inability to fly helicopters is an indictment of my fixed-wing airmanship; not coping well with notoriously perverse edge cases outside your domain of expertise isn't inherently a sign of failure (it's not a licence to stop improving, of course). Brains and 6502s are quite literally entirely different kinds of computing, much like designing for FPGA is weird and different from writing x86 assembly or C.
A far more interesting question is "could a neuroscientist understand an FPGA?".
Because that's another issue, evolution is a pretty greedy algorithm and nature doesn't care if you don't understand her architecture decisions (metaphorically speaking ofc)
A key point of the article is that we can't really be sure this is the case, since the analytical tools used by neuroscience arguably wouldn't reveal this kind of structure even if it did exist.
It is also a counterargument for the paper itself: If you know so little about two different systems, than you can't take any tool from system 1 and use it on system 2 and expect any resemblence or transfer of results.
Yes we are not 100% sure that there is somewhere a fetch decode execute loop but right know with the knowledge we have, it is unrealistic.
Why would you wan't something like this to be in our brains anyway? Its already a problem to have data an processing divided.
So in most case you can get some people to specialize and understand the simple models and create concepts and tools to understand the more complex models.
In electronic circles you still have floating around a lot of mini-integrated circuits with 20-50 transistors that are easy to understand. And you can learn to group individual transistors in mall groups that do something useful (for example limit the output current, simulate a resistors, amplification, ...)
Then you can learn to decode the intermediate models with 100-1000 transistors, and then the models with a few thousand of transistors, and then ...
So, it's very suspicious that there are no animals with a minibrain with is finite automata of 3 states.
There are also some cases where all the intermediate steps dissapared, for example the transition form prokaryote(bacteria) to eukaryote (animal, plants, protozoa, ...) And IIRC nobody understand the intermediate steps. But there are some clues, many structures are shared between prokaryotes and eukaryotes, mitochondria are probably trapped bacteria (they have their own DNA and too many external membranes, ...)
What is your basis for saying that no such animals exist? Exactly how many states there are depends a great deal on the level of analysis. How many states does a 6502 have? At the physical level, an enormously large (possibly even infinite) number. At the level of analysis appropriate for programming one, considerably fewer.
I also think the brain has important features more detailed in function than a 6502 so the applicability of the methods isn't necessarily invalid. For example, we don't know why neurons can transfer mitochondria between them, we don't know what is encoded in DNA, we don't even know how instinctive behaviors are encoded (or perhaps I just don't know).
As someone who once reverse engineered a processor from the gates up, I honestly don't know how one could do it top-down. The details at the bottom are critical, but also not critical. Decoding the microcode was critical, but determining precise timing and some other things was not needed to write a fairly functional emulator.
I like this question. I'm not a neuroscientist though, but I think most neurons come in fibers that contain several of them. So even if a certain connection was critical, a single one dying will still leave a functional fiber. Because neurons are plastic (to a limited but significant extent), any effect from that single neuron can be partially or completely compensated.
The orders of complexity involved alone dwarf any other comparison you might want to make and that's before we get into the effects of nurture, environment, interaction, education and so on.
I have really issues even accepting that anyone would even try something like this and get it published. Also the conclusion of this paper is broken.
"In other words, we asked if removed each transistor, if the processor would then still boot the game. Indeed, we found a subset of transistors that makes one of the behaviors (games) impossible. We can thus conclude they are uniquely necessary for the game—perhaps there is a Donkey Kong transistor or a Space Invaders transistor. "
A fantastic comment to show that describing a system is not the same as understanding the system!
An example. Early research which lesioned the hippocampus, lesions brought about impairments on memory tasks...but not all memory tasks. Memory of individual items, or feeling of familiarity without recollection seemed to be relatively preserved. Particularly affected however, were memories involving relations between pairs of items...but not always...those item pairs could be remembered by constructing a story about them, or making one item a feature of another item..so it seemed that item relations that were arbitrary were particularly affected by lesions, and showed more "activity" in neuroimaging studies. and so on. The hippocampus seems to fulfilling a role of binding high-lvel percepts into memory traces for which there is not some lawful/generalizable relation. This was a broad over view..but this goes beyond characterizing a brain region a "donkey kong".
Only there's no transistor uniquely responsible for moving stuff up and down. There are transistors responsible for particular bits of output of particular machine code commands, like ADD or MOV. But they are commonly used by almost all the code, so the most probable difference between code triggering the error and a code that's working correctly - would be "how big values we're working on", and if the values are in the "correct" range - that transistor will make a difference. It very well might be that x coordinates are big and y coordinates are low, so it's working for y but not for x. In that particular level of that particular game, assuming the memory was in a particular state before you started that game.
The problem lies in trying to assign too high-level meaning to stuff that works on much lower level of abstraction. It's very similar to alchemy or astrology. Searching for correlation between unrelated events and basing elaborate theories on that.
Aside from that, I object to the view point more generally. Can one not understand something about how cultures work without knowing how brains work? Can one not know something about how an ape behaves without understanding their gut-biome? I just disagree with the view that the only worthwhile understanding of phenomena is from the bottom...that there is some true level of description at which something must be understood.
Consider the following pseudo code:
i <- 0;
total <- 0;
while(i < 10) {
total <- total + i
i <- i + 1 }
output total
At the most naive it would actually use registers and jump and compare instructions to do the loop.
A different approach would do "loop unrolling" to avoid any jump instructions, instead applying sequentially the addition and comparisons.
The most aggressive approach would simply replace this whole program with
output 45
Absolutely this. An explanation of evolution in terms of atomic interactions would lack some of the explanatory and predictive power we gain at the higher levels between bio-chemistry and phenotype. I think part of good science (and FWIW software design) is choosing the correct abstraction level, the one with the best predictive, descriptive and explanatory power. Sometimes these will be different abstractions or models that have different affordances.
In the case of the OP I think it's easy to underestimate how hard it is to generate a bottom up explanation of a processor given that we already have such an explanation from its status as engineered artefact. The answer is simple if you know the answer.
Peter Watson's book "Convergence" is a delicious description of this broad phenomenon in scientific investigation(s).
Yes, you can, but that's psychology or sociology, not neuroscience. The goal was to understand brain, and be able to replicate it or modify it. Understanding just the behaviour is like understanding that pressing "fire" will shoot in Space Invaders. It's something, but you don't need to dissect a CPU to know that, and it doesn't move you closer to creating your own CPU or modifying this one.
> the only worthwhile understanding of phenomena is from the bottom
That's misrepresentation of my point. I wrote that in the CPU example neuroscience tools failed because of missing the abstraction levels between transistors and behaviours.
> there is some true level of description at which something must be understood
There is optimal level of description, yes. And notation makes a huge difference.
When we try to understand deep learning, do we try to understand it at the level of individual transistors, at the level of a 'neuron', at layers, or at the level overall design? Well it depends, right?
Maybe there really is no software level between hardware level and behaviours, but it seems unlikely to me, because I can think about abstract ideas using that brain.
The reason why you can say that one transistor doesn't handle moving up and down is because we can engineer and economically produce systems with enough spare resources and performance to implement general computing devices capable of running arbitrary programs.
If instead you were playing say... a very early arcade game, you might very well actually have the 'X coord' register implemented in a fixed discrete component.
One of the assumptions baked into biology is that by the time you're talking about cells, you're talking about implementations of behavior subject to reasonably strict constraints (plus hilarious amounts of path dependent evolution), with the assumption that things are more like old school analog/digital electronics where every component counts, versus modern integrated processors.
That's not to say that they have the perfect analytical techniques for solving problems in their domain space, or any domain space.
She was a computer science major and A student, but her model of how computers worked was way off.
However, I think this misses part of the point. We use these methods because we have very little data available. There are tons of interesting new ways to analyze brain data that I think computational neuroscientists are dying to explore, but don't have enough data to do so. If we had a lot more data, we might not be using these approaches.
Why wait until you have the best data if there seems to be a lot of low hanging fruit you can get now? It could be wasted effort, but if it does work wouldn't you be more confident on real data?
Even if it doesn't prove to be relevant, this type of statistics seems very interesting and widely applicable. Obviously reverse engineering, probably identifying alien life, characterizing group behavior on the internet... anywhere where you're still discovering structure and don't know what to expect.
A big issue touched upon in this article is that the space of possible dynamical systems represented in the brain is large, and trying to collect data is not a practical way of trimming this search space. It's more useful to look at types of dynamical systems that have certain stability properties that are desirable for computation.
But the issue then becomes that these dynamical systems become mathematically intractable past a few simplified neurons. So it's really hard to make progress either by looking at data, or by studying simplified dynamical systems mathematically.
There is a third option. Evolve smart dynamical systems by large scale brute force computation. Start with guesses about neuron-like subsystems with desirable information processing properties (at the single neuron level, such properties are mathematically tractable). Play with the configurations, the rules of evolution, the reward functions, the environment, everything. This may sound a lot like witchcraft but look at how far witchcraft has taken machine learning in recent years (deep learning is just principled witchcraft). This is IMO the only way we will learn how biological intelligence works.
Perhaps neuroscience should move in the same direction: how I think cortex work? Test in under (simple) working conditions. See if it makes sense. Move to more complex working conditions. Rinse an repeat.
In actual processors math is rather useless too... beyond some niches. Even in susceptible issues such as formal verification of coherence protocol design, mathematical tools are rather limited (due to state explosion).
" But no attempt is made to analyze the similarities and differences in those behaviors. All three game behaviors rely on similar functions. Depending on the level of similarity between the behaviors, you might think of it as trying to find a lesion that only knocks out your ability to read words that start with “k” versus words that start with “s.” That’s an experiment that’s unlikely to succeed. But if the behaviors are more like “speaking” vs “understanding spoken words” vs “understanding written words” then it’s a more reasonable experiment.
The authors argue that neuroscientists make the same mistake all the time; that we are operating at the wrong level of granularity for our behavioral measures and don’t know it. That argument denies the degree to which we characterize behaviors in neuroscience, and how stringent we are about controls.
The authors point to the fact that transistors that eliminate only one behavior are not meaningfully clustered on the chip. But what they ignore are the transistors that eliminate all three behaviors. Those structures are key to the functioning of the device in general. To me, those 1560 transistors that eliminated all three behaviors are more worthy of study than the lesions that affect only one behavior, because they allow us to determine what is essential to the behavior of the system. You can think of those transistors as leading to the death of the organism, just as damage to certain parts of the brain cause death in animals."
(The very idea that it does is actually laughable, akin to medieval fantasists imagining that flying machines must have huge white wings with fleshy feathers.)
(There is a lot of work on this organism, and I haven't kept up with the literature. So maybe the comment is a bit out of date... someone more knowledgeble should chime in.)
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2904967/#__ffn_... [overview of behavioral assays at systems level]
http://conferences.union.wisc.edu/ceneuro/program-info/organ... [next major conference]
And, of course, my favorite, WormWeb:
Reading this and the comments makes me question the similarity of the fields somewhat. Perhaps it is just our tools for comprehension that are shared between the two rather than any deeply tactical, functional commonality.
To that end, I think that experts in these fields could communicate very effectively with each other once some vocabulary had been sorted out. How effective one expert would be in the other's field is less clear to me.
The obvious problem here is the clear mismatch between the behaviors and their research objectives and methods.
If they wanted to understand transistors, they'd do what cellular neuroscientists do, and isolate and manipulate individual transistors inputs and measure the outputs.
If they wanted to understand how clusters of transistors, whose activities are tightly coupled (as you'd expect them to be in a logic gate), then you'd isolate those, and manipulate the inputs and measure the outputs.
If you wanted to understand higher levels of organization, using a lesion approach, you need to decide how much to lesion. In the brain, function is localized in clusters of related activity, and there is usually a lot of redundancy. Single neuron lesions are not usually enough to have noticeable effects. But even then, a lesion approach is more interesting when you couple it with real experiments. Consider this paper by Sheth et al. https://www.nature.com/articles/nature11239, which had subjects perform a cognitive control task before a surgical lesion to the dorsal anterior cingulate, coupled with single unit recordings, and then had them perform the same task after the lesion. The experiment yielded pre-lesion behavioral and neural evidence of a signal related to predicted demand for control, and post-lesion, the behavioral signal was abolished.
Of course, the Sheth paper would not have been possible without the iterative improvements in understanding made by prior work, including Botvinick's neural models of conflict monitoring and control. That is, its iterative; and this cpu paper was never intended to be iterative.
Following links in through "code and data":
http://ericmjonas.github.io/neuroproc/pages/data.html
I found:
https://github.com/ericmjonas/neuroprocdata
But I couldn't find any link to the c++ code. Surely the emulator is also needed in order to be able to reproduce the research?
A bit of a shame they used closed source games - I'm not sure how one would go about obtaining copies (legally). But it would be interesting to try replication via other places/demos - as they only model booting anyway.
I submit that this direction is an important one to pursue.
It reminds me of talking to another player in an online Risk game where they didn't understand what an AI player was trying to do. The code was open source and something like only three functions, but in practice they did something completely different.
A CPU has so much hardware common to most instructions that any failure will take it down completely. That's less true of a GPU, where a failure of one of the massively parallel units is likely to manifest as some alteration of the output image.
Creating a "lesion" in the motor control boards would effect behavior of the attached devices, as well as, perhaps the output on the LCD. Similarly for the sensor boards.
Once the "lesions" have let the neuroscientist determine the function of the peripherals, they could look at the effect of lesions in the microprocessor on the functions of the peripherals and system as a whole when running various programs.
Maybe a program that exercises the "left side" motors, a program that exercises both sides, etc.
Maybe a microprocessor alone is too small of a unit of functionality, akin to studying an amygdala in a petri dish.
A CPU is founded on a limited set of basic components that possess reasonable qualities, behave consistently, and only scale to large quantities with identical repetition.
Just leave out the deeper materials science and solid state quantum physics behind the "why" of how transistors operate.
Using the same analytical techniques against a cpu who's design is unknown at time of analysis. Nice meta-analysis, clickbait title.
Edit: reformatted, thanks.
> This suggests current analytic approaches in neuroscience may fall short of producing meaningful understanding of neural systems, regardless of the amount of data.
Srsly a CPU has nothing to do with a brain at all. It doesn't make sense to use technics from one for the other.
I have no idea how anyone comes up with such an idea and even publishes it.
A Brain itself is everything. Ram and CPU.
A CPU is just a CPU there is no state in physical form.
A CPU is a turing machine, a brain isn't.
I'm not a neuroscientist (I cannot afford medical education), nor am I a microprocessor engineer (yet). But I understand how systems work, so I might have a chance to understand how neural networks work (as models and their real counterparts) and I might have already an understanding on how microprocessors are designed by principle. So, yes, a neuroscientist who decides to visit some lectures on digital logic circuits and microprocessor design might have a chance to understand it! I'm really confused about this quenstion.
From the abstract:
> here we take a classical microprocessor as a model organism, and use our ability to perform arbitrary experiments on it to see if popular data analysis methods from neuroscience can elucidate the way it processes information. Microprocessors are among those artificial information processing systems that are both complex and that we understand at all levels, from the overall logical flow, via logical gates, to the dynamics of transistors. We show that the approaches reveal interesting structure in the data but do not meaningfully describe the hierarchy of information processing in the microprocessor.
The idea is to apply the modern neuroscience approach to a microprocessor to see what level of understand of the microprocessor is extracted. tl;dr: The high-level "meaning" of the processor's design is not extracted.
The purpose seems to be to examine the inherent limitations of modern neuroscience by applying it to a design that we do understand quite well apart from neuroscience, something we ourselves designed.