Generative adversarial networks for reconstructing natural images from brain activity

We explore a method for reconstructing visual stimuli from brain activity. Using large databases of natural images we trained a deep convolutional generative adversarial network capable of generating gray scale photos, similar to stimuli presented during two functional magnetic resonance imaging exp...

Celý popis

Uloženo v:

Podrobná bibliografie
Vydáno v:	NeuroImage (Orlando, Fla.) Ročník 181; s. 775 - 785
Hlavní autoři:	Seeliger, K., Güçlü, U., Ambrogioni, L., Güçlütürk, Y., van Gerven, M.A.J.
Médium:	Journal Article
Jazyk:	angličtina
Vydáno:	United States Elsevier Inc 01.11.2018 Elsevier Limited
Témata:	Adult Brain Brain - diagnostic imaging Brain - physiology Brain mapping Datasets fMRI Functional magnetic resonance imaging Functional Neuroimaging - methods Generative adversarial networks Humans Image Processing, Computer-Assisted - methods Magnetic Resonance Imaging - methods Models, Theoretical Neural Networks (Computer) Neuroimaging NMR Nuclear magnetic resonance Pattern Recognition, Visual - physiology Perceptions Reconstruction Vision Visual stimuli Reconstruction fMRI Generative adversarial networks Vision
ISSN:	1053-8119, 1095-9572, 1095-9572
On-line přístup:	Získat plný text
Tagy:	Přidat tag Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!

Popis
Shrnutí:	We explore a method for reconstructing visual stimuli from brain activity. Using large databases of natural images we trained a deep convolutional generative adversarial network capable of generating gray scale photos, similar to stimuli presented during two functional magnetic resonance imaging experiments. Using a linear model we learned to predict the generative model's latent space from measured brain activity. The objective was to create an image similar to the presented stimulus image through the previously trained generator. Using this approach we were able to reconstruct structural and some semantic features of a proportion of the natural images sets. A behavioural test showed that subjects were capable of identifying a reconstruction of the original stimulus in 67.2% and 66.4% of the cases in a pairwise comparison for the two natural image datasets respectively. Our approach does not require end-to-end training of a large generative model on limited neuroimaging data. Rapid advances in generative modeling promise further improvements in reconstruction performance. •A generative adversarial network (DCGAN) is used for reconstructing visual percepts.•Minimizing image loss, a linear model learns to predict the latent space from BOLD.•With a GAN limited to 6 handwritten characters, detailed features can be retrieved.•Reconstructions of arbitrary natural images are identifiable by human raters.•The specific GAN is a component and replaceable by advanced deterministic generators.
Bibliografie:	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 content type line 23
ISSN:	1053-8119 1095-9572 1095-9572
DOI:	10.1016/j.neuroimage.2018.07.043