Deep image reconstruction from human brain activity (2017)
biorxiv.org
biorxiv.org
It's interesting that they were not able to extract good images from the test when they just asked the subjects to imagine the pictures. I speculate that is because the vast majority of people are able to recognize images of things, but not produce those images.
For a simple example, see this article about people trying to draw a bicycle. Before clicking, try to draw a 2D representation of a bicycle -- the wheels, the frame, pedals, handlebars, etc. You'd be surprised how difficult it is. https://road.cc/content/blog/90885-science-cycology-can-you-...
I would be very interested to see them repeat the experiment on e.g. artists who specialize in photorealism, or asking an expert on a very particular thing to imagine that particular thing (e.g. ask a violinist to imagine a violin). I wonder if that would yield more recognizable results than asking a random grad student to visualize a lion's face.
https://books.google.com/books?id=_gA_DwAAQBAJ&pg=PT87&lpg=P...
I was a kid growing up in Far Rockaway, I had a friend named Bernie Walker. We both had “labs” at home, and we would do various “experiments.” One time, we were discussing something — we must have been eleven or twelve at the time — and I said, “But thinking is nothing but talking to yourself inside.
“Oh yeah?” Bernie said. “Do you know the crazy shape of the crankshaft in a car?”
“Yeah, what of it?”
“Good. Now, tell me: how did you describe it when you were talking to yourself?”
So I learned from Bernie that thoughts can be visual as well as verbal.
which leads me to wonder if the capability exists on a spectrum and is stronger or weaker in some. A stronger capability might correlate to a stronger 'signal' ...?
I can close my eyes and think of something, but I won't perceive anything more than a linear description of it.
At best, I can use these visual descriptions to draw an outline in my head, but it instantly fades out if I don't continuously re-draw it.
Kind of like this: https://i.imgur.com/0zuPIPV.gifv
- Can you imagine the color of the eyes and hairs of your coworkers?
- Can you imagine the home screen of your phone and see how many application icons can fit in a row and column?
- Can you imagine Google's logo and see the color of each letter?
- Can you imagine your car manufacturer's logo and draw it?
2) Hair for all of them, but I can only picture the eyes of the people who had unusual colors (blue, gray, green). Maybe I never paid much attention if they had normal brown eyes.
3) Row yes, but I wasn't sure about the column (4 or 5).
4) Yes.
5) easiest task on the list.
I can't help but be a little sceptical of the concept though. I believe there's probably a spectrum but maybe not as wide is claimed. Based on the comment sections of basically any article on the subect, I wonder if there isn't something like imposter syndrome going on. Where the majority of people dont think they have as sharp mental imagery as the average person.
I thought this was how everyone’s mind worked, until I met my ex, who is on the opposite end of the scale. She described her inner world as sometimes more vivid and detailed than the real world. I thought she was almost unique in this, until I started asking other people and realised that most people see images in their mind, of varying detail and vividness. Some can easily project onto their visual input and, for example, put a moustache or wings on someone. I’ve also come to learn that many people hear their own voice when they think (and many, non-schizophrenic, people hear other voices too), and can also imagine smells and tastes.
My episodic memory is affected by my aphantasia, as well as my people recognition skills. I can find it hard to recognise someone if they’ve changed their hair, grown a beard or changed their appearance in some other way.
I do dream though, vivid dreams, and am quite good at recalling them. But again the memory of the dreams, like other memories, is not visual.
I’m quite fascinated with this subject, and have some thoughts of picking up where I left off on my psychology master to be able to study it more. It’d be interesting to see if there are correlates in big-5 personality, iq, mental health etc.)
Until I read [0], I had never thought about it and just assumed that everything else was like me. Hah!
[0] https://www.lesswrong.com/posts/baTWMegR42PAsH9qJ/generalizi...
Yes, and Forbidden Planet "plastic educator" comes to mind :)
I've noticed though that as I've spent years learning Japanese characters, I am somewhat better at visualizing them in my head. I can see (or "feel", as it's not really visual exactly in the same sense as seeing, although they are not just concepts but have spatial locations) whole subcomponents at a time instead of having to plot each line.
See: image streaming by Win Wenger http://winwenger.com/imstream.htm
I'd be very curious to see if you are able to see simple things in your mind's eye after practising this for a week or two
I challenge anyone who believes they have a highly developed mind's eye to draw such a face purely from memory.
Weirdly I’m actually quite good at drawing something from memory like a bicycle (I love Pictionary!) but I can’t actually make a image of it in my head.
Brains are strange.
Never heard of this "condition" before. It may apply to me. I can somewhat imagine a static thing or a face but trying to "zoom in"/"rotate" or imagining smaller details completely ruins the illusion.
However, if you are using an fMRI, as in the paper, all you need is the accurate representation somewhere (and you need to know where that somewhere is of course).
Another example: let's say you look at a picture of cheetah. Now you try to imagine it; what appears to you is a vague sort of resemblance but nowhere near as clear as when you were looking directly at it. But, just because the prompting for you to imagine the image didn't cause an accurate recreation in consciousness, doesn't mean it didn't occur elsewhere, perhaps at some earlier/lower-level stage of visual processing.
For example, I cannot visualize a 13-sided polygon accurately, but I can draw one pretty easily without having any drawing skills.
A more realistic problem in the case of the bicycle drawing has to do with the procedural knowledge they use for drawing. For instance, maybe they approach it analytically, top-down, breaking it into major components to draw first, and those into sub components. Well, in order to do that, they need explicit knowledge of that hierarchy; they may have an accurate representation of the image somewhere, but they can't bring it directly into consciousness, and they didn't break it into that hierarchy while they had it around. So now what's left to them is to attempt to produce the hierarchy from an imperfectly recalled image, which leads to the strange configurations.
Just a guess—but I wouldn't be surprised to learn it's something on those lines.
It seems reasonable to me. To use a computer as an analogy, the set of signals produced by a webcam recording a video to a hard drive are not going to be the same as the ones of the video playing from the hard drive on to the screen.
Some people may be able to produce a bicycle in more details than others but that depends on how developed and healthy their brain is to use their executive functions. Without knowing all of the subjects medical history—early dementia, strokes, all the good stuff- and current health, it’d come as no surprise that even consciously we cannot produce true representations of what we saw.
Police sketches from eye witnesses may be a good thing to look at when thinking of producing images from memory. Some people have better brain health and are able to utilize many different high and low-level functions to produce more accurate descriptions. Others not so much.
But to keep it simple: read, do puzzles, stay active, do not drink or do drugs heavily. ——
Everybody already has a genetic makeup that limits their health—especially the brain—to a preset level. Some people suffer from infantile amnesia—basically not remembering early developmental stages (2-4) and preadolescent (~10 y/o); other’s claim to remember their birth (which is silly since ability to form memories is 2 y/o+). Some have the luxury of having strong working memory^1—a fancy word for temporary memories—but suffers from lackluster short-term memory.
I’m sure you get the point I was trying to make with that mediocre explanation.
tldr: You can not change your brain to not have dementia if your genetic makeup says so. You can however delay the onset possibly by keeping your brain active on a daily basis.
Have you spoke about it at great lengths when older? Is it often brought up about how they took the cast off? If so, you’re looking at the idea of explicit (declarative) memory—the storing of facts and events. If over the years you are constantly reminded of this event, it is more than likely that you’re trying to generate a (possible semi-accurate) image or memory of that particular event.
As far as your second comment goes, mother tongue and languages are different all together. When you learn as a yongue age, your cerebral cortex is still developing and your mind is absorbing everything like a sponge. As you learn these words, continue to use them, that language and those procedures are going to get stored in your long-term memory and then you’ll be able to recite all of these words and form sentences without thinking twice—something we would call implicit memory which helps us with procedure memories (riding a bike, writing, etc.).
https://www.biorxiv.org/content/biorxiv/early/2017/12/28/240...
https://github.com/KamitaniLab/DeepImageReconstruction
The code is well-commented Python and a pleasure to read.
Bottom line, I'm definitely interested in this paper, but want to wait and see if others can replicate the results before getting overly excited.
A world in which "you have nothing to hide" turns into "you have nothing you can hide" is immensely disturbing.
I bet they will soon replace the search bar by just a button.
If I were a billionaire, one of the projects I'd do would be to apply this sort of technology at scale, upon volunteers who were witness to poorly recorded historic events. WWII, Beatles concerts, JFK assassination, 9/11... all of them have been recorded from angles/perspectives that exist only in peoples' memories, and the longer we wait, the more of those memories will be lost.
Here's a Guardian article about it: https://www.theguardian.com/science/2016/apr/27/brain-atlas-...
And a link to the paper: https://www.nature.com/articles/nature17637
The obvious next step is to combine that with the findings in the article.
Ie: Create a VR (or AR) headset that can both present an audio-visual experience to the user, WHILE having enough sensors to accurately measure the users brain state (and other bodily states, such as heart rate, blood pressure, etc).
That will place a game developer in a position to predict what events a game can create that would create the greatest possible engagement in a given gamer. They can time rewards, challenges, esthetical or sexual stimuli in the way that generates the greatest possible control of the player's future behaviour.
In the hands of the greatest data scientists, this will allow them to create a hithertdo unprecedented level of addiction, which can then be monetetized through subscriptions, micro transactions, embedded adverticements or just by selling the data.
In the hands of governments, this can be used to pacify any population, drive behaviour in desired directions, or create support (even demand) for given policies.
In the hands of parents, it can greatly help them teach their children whatever they want to teach them. (Be it morality, religion, self-discipline, or whatever)
In the hands of a great psychologist, it would be able to cure most mental disorders less severe than outright psychosis.
How will we face this? Will anyone outside the elite be able to prevent this being used against them?
> We used the Caffe implementation (Jia et al., 2014) of the VGG19 deep neural network (DNN) model (Simonyan & Zisserman, 2015; available from https://github.com/BVLC/caffe/wiki/Model - Zoo) . All visual images were resized to 224 × 224 pixels to compute outputs by the VGG19 model . The VGG19 model consisted of a total of sixteen convolutional layers and three fully connected layers.
They used a VGG-19 network for the feature detection, so in my basic understanding the results of this could already be improved by switching to a ResNet.
* ResNets were better at feature extraction for image clustering [0]
* One of the trade-offs of ResNets seems to be their relative complexity to VGG [1]
* It surprises me that ResNets aren't significantly faster to train based on the large reduction of FLOPs (from the ResNet paper VGG-19 had 19.6 billion FLOPs vs ResNet-34 with 3.6 billion FLOPs) - I think people just train deeper ResNets e.g. ResNet-50
[0]: https://medium.com/@franky07724_57962/using-keras-pre-traine...
[1]: https://www.reddit.com/r/MachineLearning/comments/6e6mlf/d_i...
https://distill.pub/2018/differentiable-parameterizations/#s...
They discuss VGG vs non-VGG architectures in the context of style transfer in Section 2, which was interesting to me.
> One thing they noticed was that using features from a pretrained ImageNet VGG-16/19 CNN from 2014 (4 years ago), like the original Gatys paper did, worked much better than anything else; indeed, almost any set of 4-5 layers in VGG would provide great features for the style transfer optimization to target (as long as they were spread out and weren't exclusively bottom or top layers), while using more modern resnets (resnet-50) or GoogLeNet Inception v1 didn't work - it was hard to find sets of layers that would work at all and when they did, the quality of the style transfer was not as good. Interestingly, this appeared to be true of VGG CNNs trained on the MIT Places scene recognition database too, suggesting there's something architectural going on which is not database specific or peculiar to those two trained models. And their attempt at an upscaling CNN modeled on Johnson et al 2016's VGG-16 for CIFAR-100 worked well too.
[0]: https://www.reddit.com/r/MachineLearning/comments/7rrrk3/d_e...
I guess it makes sense that HN is skeptical about results more closely related to the community's expertise but it's a pretty crazy leap from being able to reconstruct something vaguely similar to what a person is currently seeing vs. trying to read someone's mind. I still have seen very little evidence that such a thing could at all be possible with FMRI. The spatial and temporal resolution are just far too low. The more everyone is impressed with results like this the more we delay the hard work of developing tools that actually have a shot at doing something like that.
Among useful applications I would imagine, for example, the fast image-based information retrieval - e.g. useful for recommendation systems (in other words - Netflix could invest in a 'imagine anything and we find you a movie like that').
But I think the greatest utility of this tech is not in the movie industry, but in the gaming industry.
Imagine a photo realistic version of sims where your family and friends know EXACTLY how to react to maximize your sense of meaning, happiness, enjoyment in precisely the way that is needed in order to prevent you from leaving the game?
This is as bad as the dead salmon results.
EDIT: You can flag all you want, but look at this playback of the reconstruction:
https://video.twimg.com/tweet_video/DSrWxhSVQAE2Xix.mp4
There's nothing useful happening here.
Don't be surprised when it "finds" the image, pulled from raw static that contained no signal.
> While our model was solely trained with natural images, our method successfully generalized the reconstruction to artificial shapes, indicating that our model indeed reconstructs or generates images from brain activity, not simply matches to exemplars.
If I am reading that correctly it is reconstructing images that the subjects had not seen before.
https://github.com/KamitaniLab/DeepImageReconstruction
Not a hoax.
Your comparison to Deep Dream is quite apt, for good reason that you clearly don't realize-
Deep Dream is an attempt to "enhance patterns in images via algorithmic pareidolia"- Pareidolia being a "psychological phenomenon in which the mind responds to a stimulus". So yes, this project is affected by a real life "deep dream".