Reconstructing natural images from human fMRI by alternating encoding and decoding with shared autoencoder regularization

•We proposed the mutual promotion of visual encoding and reconstruction models.•We designed alternating optimization bassed on seim-supervised learning.•We devised inter-sample differentiated representations to augment small dataset.•Our proposed model achieved state-of-the-art reconstruction perfor...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Biomedical signal processing and control Jg. 73; S. 103397
Hauptverfasser: Qiao, Kai, Chen, Jian, Wang, Linyuan, Zhang, Chi, Tong, Li, Yan, Bin
Format: Journal Article
Sprache:Englisch
Veröffentlicht: Elsevier Ltd 01.03.2022
Schlagworte:
Alternating optimization
fMRI
Shared autoencoder
Visual encoding
Visual reconstruction
fMRI
Visual encoding
Shared autoencoder
Alternating optimization
Visual reconstruction
ISSN:1746-8094, 1746-8108
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:•We proposed the mutual promotion of visual encoding and reconstruction models.•We designed alternating optimization bassed on seim-supervised learning.•We devised inter-sample differentiated representations to augment small dataset.•Our proposed model achieved state-of-the-art reconstruction performance. Reconstructing the viewed natural images from the corresponding functional magnetic resonance imaging (fMRI) of human visual cortices is extremely difficult. Utilizing deep learning techniques, the quality of reconstructed images can be significantly improved. This however is subject to availability of sufficient number of pair samples, which is not currently the case. In this study, we propose to perform alternating encoding and decoding to realize the mutual promotion of both based on shared semi-supervised learning and accomplish better reconstruction of natural images from the corresponding fMRI voxels. In our proposed method, the encoder and decoder are used to respectively map the images to the fMRI voxels (visual encoding), and the fMRI voxels to the images (visual reconstruction). We argue that combining the encoder and decoder in different sequences can form two converse and shared autoencoders to regularize the supervised learning of both through the unsupervised learning of fMRI voxels and images. More importantly, we alternatingly train the encoder and decoder with shared autoencoder regularization. Here, the premise is that a better encoder can produce a better decoder, and vice versa. The pixel-level identification of the proposed method achieves up to 89.5%, which indicates at least 2% superiority and is the new state-of-the-art compared to the previous works in terms of the image reconstruction performance.
ISSN:1746-8094
1746-8108
DOI:10.1016/j.bspc.2021.103397