Deep image reconstruction from human brain activity [pdf] | Hacker News Reader

Deep image reconstruction from human brain activity [pdf]

biorxiv.org

178 pointsgrzm34 comments

They trained a deep neural network to reconstruct what a patient was seeing from fMRI signals and were mildly successful. Cool!

Things get really awesome in Figure 4. They had patients imagine an image/symbol that that they would then try to reconstruct from fMRI signals. They had a few successes and were able to operate at better than chance when humans evaluated the reconstructed shapes. That's awesome!

http://news.berkeley.edu/2011/09/22/brain-movies/

Guess the kyoto group are competing w/ Jack Gallant's lab...

btw, pycortex from his lab is out on GitHub...neat to play around w/ if you have the fMRI data

But was the network trained on the same individual as it was used on?

Or is it possible to train it once and then apply it to any random person?

the former

Cool results. Just want to point out that there's a time lag between "thoughts" and the appearance of a signal through fMRI, which explains why the image needed to be imagined for a long period of time (8 seconds).

Most thoughts are more fleeting than this, and couldn't be read with fMRI. So there's no need to get /too/ concerned about the privacy of our thoughts. Not yet :)

> Each imagery block consisted of a 4-s cue period, an 8-s imagery period, a 3-s evaluation period and a 1-s rest period.

Strap someone to a gurney and you have all the time in the world. Would an fMRI require a search warrant?

You can't make them hold the image in their head for you to pick it out, though.

You'd also need to make them keep their head very still.

There are substances that block voluntary muscle movements. Pancuronium bromide, for example.

Don't think of the pink elephant.

I can't imagine pictures in my head, so regardless of how long I will think about pink elephant, you will see nothing.

For people like you, they can always fall back to the old "$5 wrench" method...

IMHO, thermorectal cryptoanalysis with soldering iron is much more effective for similar price.

OK. I'll just think about something else. I don't know why people think tricks like this work on anyone but the extremely simple-minded.

I don't know how your brain works, but for most people just hearing/reading the words "pink elephant" will cause their System 1 to immediately bring that concept into their consciousness.

This isn't "simple minded" it's just how brains work, and the alternative of having to actively think about a word to remember what it means, would be much slower.

I wonder what would happen if you could feed back the output into the visual system using VR in near real time. Would you be able “see” anything you could think of?

And then feed the signal from that back into... oh fuck it

>Conjures vague, smudgy figures and vistas passable as abstract thought but ridiculous in realization

I want a better neural GPU!

Very impressive. Please note this was done with magnetic resonance imaging.

A more invasive approach was this: "Reconstruction of Natural Scenes from Ensemble Responses in the Lateral Geniculate Nucleus" (1999).

- http://www.jneurosci.org/content/19/18/8036

- http://www.jneurosci.org/content/jneuro/19/18/8036/F2.large....

What is a bit concerning are the applications, e.g: forcibly extracting thoughts from non-collaborating people.

Seems like a weird form of overfitting.

You give the brain a set of images to fit/train the original model. Then you show the brain a set of images similar to the ones used to train the model.

... you will never really understand what the brain sees in novel situations unrelated to the training task.

Here's a related article: http://www.sciencedirect.com/science/article/pii/S0960982211... ...and a few other interesting ones from my link archive

Brain map http://www.extremetech.com/extreme/227498-uc-berkeley-team-b...

3d-printing by thought http://www.bbc.com/future/story/20130613-3d-printing-your-th...

Brain computer interface for controlling a Hex Bug and painting tools https://en.wikipedia.org/wiki/OpenBCI

Typing by thought https://spectrum.ieee.org/the-human-os/biomedical/bionics/fa...

What is up with "These authors contributed equally to this work."?

Often, first author is considered the primary contributor, while last author might be considered a principal or adviser. By adding a disclaimer and listing authors alphabetically, you can try and dispel that misconception if this isn't the case for your paper.

Typically most credit goes to first and last author. However, there are cases where one person needs to be first author, but multiple people are equally deserving.

This strongly depends on what field you are in and can vary even between minor sub-disciplines. Also that statement is frequently meaningless and authors get added for politics target than contribution.

One step closer to Black Mirror Season 4: Crocodile, among other things.

I would definitely buy a play back machine in the future so I don’t have to take so many pictures on trips.

https://www.youtube.com/watch?v=SjbSEjOJL3U

Now if only the information from EEG was half as useful from fMRI.

Or half as useful as brain implants.

Theoretically, you could do the same thing with EEG via source localization techniques[0] on the EM spectrum.

In my experience, barely any labs have the know how to implement them in a performative manor to pull something like this off whereas a lot of the fmri software is made from the manufacture (GE, Siemens, etc) that transforms the raw data into the voxels that were used in this setup.

[0] http://www.scholarpedia.org/article/Source_localization

Sounds like you still need too many electrodes !

Depends on which brain regions you want to acquire signals from (usually based on the literature), but generally yes, you need to sample from a "sufficient" number of places (one of my fmr colleges has a lead reduction paper coming out this year that will talk about this).

But keep in mind here, using eeg (and beamforming) is still way cheaper and accessible than any fmri machine, though as far as having the skills to implement such, I think it is out of the reach of most labs who do this sort of research.

Electrodes are even getting better[0] wrt cost and actual ease of real world usage (dry/non gel needed/comfort).

[0] http://doc.ml.tu-berlin.de/bbci/publications/GroVoiFaz11.pdf

Reply on news.ycombinator.com