Reconstructing natural images from human fMRI by alternating encoding and decoding with shared autoencoder regularization
•We proposed the mutual promotion of visual encoding and reconstruction models.•We designed alternating optimization bassed on seim-supervised learning.•We devised inter-sample differentiated representations to augment small dataset.•Our proposed model achieved state-of-the-art reconstruction perfor...
Gespeichert in:
| Veröffentlicht in: | Biomedical signal processing and control Jg. 73; S. 103397 |
|---|---|
| Hauptverfasser: | , , , , , |
| Format: | Journal Article |
| Sprache: | Englisch |
| Veröffentlicht: |
Elsevier Ltd
01.03.2022
|
| Schlagworte: | |
| ISSN: | 1746-8094, 1746-8108 |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| Abstract | •We proposed the mutual promotion of visual encoding and reconstruction models.•We designed alternating optimization bassed on seim-supervised learning.•We devised inter-sample differentiated representations to augment small dataset.•Our proposed model achieved state-of-the-art reconstruction performance.
Reconstructing the viewed natural images from the corresponding functional magnetic resonance imaging (fMRI) of human visual cortices is extremely difficult. Utilizing deep learning techniques, the quality of reconstructed images can be significantly improved. This however is subject to availability of sufficient number of pair samples, which is not currently the case. In this study, we propose to perform alternating encoding and decoding to realize the mutual promotion of both based on shared semi-supervised learning and accomplish better reconstruction of natural images from the corresponding fMRI voxels. In our proposed method, the encoder and decoder are used to respectively map the images to the fMRI voxels (visual encoding), and the fMRI voxels to the images (visual reconstruction). We argue that combining the encoder and decoder in different sequences can form two converse and shared autoencoders to regularize the supervised learning of both through the unsupervised learning of fMRI voxels and images. More importantly, we alternatingly train the encoder and decoder with shared autoencoder regularization. Here, the premise is that a better encoder can produce a better decoder, and vice versa. The pixel-level identification of the proposed method achieves up to 89.5%, which indicates at least 2% superiority and is the new state-of-the-art compared to the previous works in terms of the image reconstruction performance. |
|---|---|
| AbstractList | •We proposed the mutual promotion of visual encoding and reconstruction models.•We designed alternating optimization bassed on seim-supervised learning.•We devised inter-sample differentiated representations to augment small dataset.•Our proposed model achieved state-of-the-art reconstruction performance.
Reconstructing the viewed natural images from the corresponding functional magnetic resonance imaging (fMRI) of human visual cortices is extremely difficult. Utilizing deep learning techniques, the quality of reconstructed images can be significantly improved. This however is subject to availability of sufficient number of pair samples, which is not currently the case. In this study, we propose to perform alternating encoding and decoding to realize the mutual promotion of both based on shared semi-supervised learning and accomplish better reconstruction of natural images from the corresponding fMRI voxels. In our proposed method, the encoder and decoder are used to respectively map the images to the fMRI voxels (visual encoding), and the fMRI voxels to the images (visual reconstruction). We argue that combining the encoder and decoder in different sequences can form two converse and shared autoencoders to regularize the supervised learning of both through the unsupervised learning of fMRI voxels and images. More importantly, we alternatingly train the encoder and decoder with shared autoencoder regularization. Here, the premise is that a better encoder can produce a better decoder, and vice versa. The pixel-level identification of the proposed method achieves up to 89.5%, which indicates at least 2% superiority and is the new state-of-the-art compared to the previous works in terms of the image reconstruction performance. |
| ArticleNumber | 103397 |
| Author | Yan, Bin Zhang, Chi Chen, Jian Qiao, Kai Wang, Linyuan Tong, Li |
| Author_xml | – sequence: 1 givenname: Kai surname: Qiao fullname: Qiao, Kai email: qiaokai1992@gmail.com – sequence: 2 givenname: Jian surname: Chen fullname: Chen, Jian – sequence: 3 givenname: Linyuan surname: Wang fullname: Wang, Linyuan – sequence: 4 givenname: Chi surname: Zhang fullname: Zhang, Chi – sequence: 5 givenname: Li surname: Tong fullname: Tong, Li – sequence: 6 givenname: Bin surname: Yan fullname: Yan, Bin email: ybspace@hotmail.com |
| BookMark | eNp9kMtKAzEUhoNUsFVfwFVeYGoyk7mBGyleChWh6DpkkpM2ZZopSUapT2-mrRsXXeUk_F84_zdBI9tZQOiOkikltLjfTBu_k9OUpDQ-ZFldXqAxLVmRVJRUo7-Z1OwKTbzfEMKqkrIx2i9BdtYH18tg7ApbEXonWmy2YgUea9dt8brfCov123KOmz0WbQAXY0MarOzUMAirsILT5duENfZr4UBh0YfukAKHHaz6VjjzE-HO3qBLLVoPt6fzGn0-P33MXpPF-8t89rhIZEZISDJdgmB1U2iWS6JYCWWeqZo1DWE6V1LmmmaxtmKpSnOV1SqnWulCpbWQMZ5do_T4r3Sd9w4037nYzu05JXyQxzd8kMcHefwoL0LVP0iacFg7OGHa8-jDEYVY6suA416aqACUcSADV505h_8Cwp-Qbg |
| CitedBy_id | crossref_primary_10_1016_j_jneumeth_2024_110269 crossref_primary_10_3390_brainsci14030234 crossref_primary_10_1016_j_bspc_2024_107110 crossref_primary_10_1109_TNSRE_2023_3283405 |
| Cites_doi | 10.1371/journal.pcbi.1006633 10.1109/CVPR42600.2020.00698 10.1007/s11263-015-0816-y 10.1162/NECO_a_00047 10.3390/app9224749 10.1016/j.neucom.2019.12.130 10.1016/j.neuroimage.2018.07.043 10.1016/j.neuron.2008.11.004 10.32470/CCN.2018.1031-0 10.1016/j.cub.2011.08.031 10.1523/JNEUROSCI.5023-14.2015 10.1109/LGRS.2018.2799232 10.1016/j.neuroimage.2010.07.073 10.1109/IJCNN48605.2020.9206960 10.1016/j.neuron.2009.09.006 10.1038/ncomms15037 10.1016/j.neuroimage.2019.05.039 10.1145/3386252 10.3389/fnhum.2018.00242 10.1109/CVPR42600.2020.00674 10.1016/j.tics.2006.07.005 |
| ContentType | Journal Article |
| Copyright | 2021 Elsevier Ltd |
| Copyright_xml | – notice: 2021 Elsevier Ltd |
| DBID | AAYXX CITATION |
| DOI | 10.1016/j.bspc.2021.103397 |
| DatabaseName | CrossRef |
| DatabaseTitle | CrossRef |
| DatabaseTitleList | |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering |
| EISSN | 1746-8108 |
| ExternalDocumentID | 10_1016_j_bspc_2021_103397 S1746809421009940 |
| GroupedDBID | --- --K --M .~1 0R~ 1B1 1~. 1~5 23N 4.4 457 4G. 5GY 5VS 6J9 7-5 71M 8P~ AACTN AAEDT AAEDW AAIAV AAIKJ AAKOC AALRI AAOAW AAQFI AAXUO AAYFN ABBOA ABFNM ABFRF ABJNI ABMAC ABXDB ABYKQ ACDAQ ACGFO ACGFS ACNNM ACRLP ACZNC ADBBV ADEZE ADMUD ADTZH AEBSH AECPX AEFWE AEKER AENEX AFKWA AFTJW AGHFR AGUBO AGYEJ AHJVU AHZHX AIALX AIEXJ AIKHN AITUG AJBFU AJOXV ALMA_UNASSIGNED_HOLDINGS AMFUW AMRAJ AOUOD AXJTR BJAXD BKOJK BLXMC CS3 DU5 EBS EFJIC EFLBG EJD EO8 EO9 EP2 EP3 F5P FDB FIRID FNPLU FYGXN G-Q GBLVA GBOLZ HZ~ IHE J1W JJJVA KOM M41 MO0 N9A O-L O9- OAUVE OZT P-8 P-9 P2P PC. Q38 RIG ROL RPZ SDF SDG SES SPC SPCBC SST SSV SSZ T5K UNMZH ~G- 9DU AATTM AAXKI AAYWO AAYXX ABWVN ACLOT ACRPL ACVFH ADCNI ADNMO AEIPS AEUPX AFJKZ AFPUW AIGII AIIUN AKBMS AKRWK AKYEP ANKPU APXCP CITATION EFKBS ~HD |
| ID | FETCH-LOGICAL-c300t-3f7ea49b6f45c0d47e753d94bb04f5dcc5f13021d42d25d39d51fdf6d29acd473 |
| ISICitedReferencesCount | 4 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000782654300004&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1746-8094 |
| IngestDate | Sat Nov 29 07:04:14 EST 2025 Tue Nov 18 22:43:06 EST 2025 Fri Feb 23 02:40:56 EST 2024 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Keywords | fMRI Visual encoding Shared autoencoder Alternating optimization Visual reconstruction |
| Language | English |
| LinkModel | OpenURL |
| MergedId | FETCHMERGED-LOGICAL-c300t-3f7ea49b6f45c0d47e753d94bb04f5dcc5f13021d42d25d39d51fdf6d29acd473 |
| ParticipantIDs | crossref_primary_10_1016_j_bspc_2021_103397 crossref_citationtrail_10_1016_j_bspc_2021_103397 elsevier_sciencedirect_doi_10_1016_j_bspc_2021_103397 |
| PublicationCentury | 2000 |
| PublicationDate | March 2022 2022-03-00 |
| PublicationDateYYYYMMDD | 2022-03-01 |
| PublicationDate_xml | – month: 03 year: 2022 text: March 2022 |
| PublicationDecade | 2020 |
| PublicationTitle | Biomedical signal processing and control |
| PublicationYear | 2022 |
| Publisher | Elsevier Ltd |
| Publisher_xml | – name: Elsevier Ltd |
| References | VanRullen, R., and Reddy, L. (2018). Reconstructing Faces from fMRI Patterns using Deep Generative Neural Networks. LeCun, Bengio, Hinton (b0075) 2015; 521 Eickenberg, Gramfort, Varoquaux, Thirion (b0025) 2016; 152 Russakovsky, Deng, Su, Krause, Satheesh, Ma, Huang, Karpathy, Khosla, Bernstein, Berg, Fei-Fei (b0140) 2015; 115 Wen, Shi, Zhang, Lu, Cao, Liu (b0200) 2017 Shen, Dwivedi, Majima, Horikawa, Kamitani (b0150) 2018 Zhang, Y., Wu, Z., Peng, H., and Lin, S. (2020). “A transductive approach for video object segmentation”, in Horikawa, Kamitani (b0050) 2017; 8 Han, Wen, Shi, Lu, Zhang, Fu, Liu (b0045) 2019; 198 Koch, Zemel, Salakhutdinov (b0065) 2015 Nishimoto, Vu, Naselaris, Benjamini, Yu, Gallant (b0120) 2011; 21 Seeliger, Güçlü, Ambrogioni, Güçlütürk, van Gerven (b0145) 2018; 181 Miyawaki, Uchida, Yamashita, Sato, Morito, Tanabe, Sadato, Kamitani (b0100) 2008; 60 Jiang, Qiao, Wang, Zhang, Chen, Zeng, Bu, Yan (b0060) 2019; 9 Shen, Horikawa, Majima, Kamitani, O’Reilly (b0155) 2019; 15 van Gerven, de Lange, Heskes (b0185) 2010; 22 Guclu, van Gerven (b0035) 2015; 35 Shi, Gong, Ding, Tao, Zheng (b0160) 2018 Chao, Changpinyo, Gong, Sha (b0015) 2016 6949-6958. Ren, Z., Li, J., Xue, X., Li, X., Yang, F., Jiao, Z., et al. (2019). Reconstructing Perceived Images from Brain Activity by Visually-guided Cognitive Representation and Adversarial Learning. 6707-6717. Naselaris, Prenger, Kay, Oliver, Gallant (b0115) 2009; 63 Song, Shen, Yang, Liu, Song (b0170) 2018 Norman, Polyn, Detre, Haxby (b0125) 2006; 10 Arjovsky, Chintala, Bottou (b0005) 2017 Chong, Ding, Yan, Pan (b0020) 2020; 408 Zhang, C., Qiao, K., Wang, L., Tong, L., Zeng, Y., and Yan, B. (2018). Constraint-free natural image reconstruction from fMRI signals based on convolutional neural network. Misra, I., and Maaten, L.v.d. (2020). “Self-supervised learning of pretext-invariant representations”, in 6082-6091. Styves, Naselaris (b0180) 2017 Mescheder, L., Geiger, A., and Nowozin, S. (2018). Which training methods for GANs do actually converge? St-Yves, Naselaris (b0175) 2018 Güçlütürk, Güçlü, Seeliger, Bosch, van Lier, van Gerven (b0040) 2017 12. . Beliy, Gaziv, Hoogi, Strappini, Golan, Irani (b0010) 2019 Krizhevsky, Sutskever, Hinton (b0070) 2012 Wang, Yao, Kwok, Ni (b0195) 2020; 53 Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair (b0030) 2014 Naselaris, Kay, Nishimoto, Gallant (b0110) 2011; 56 Rahman, S., Khan, S., and Barnes, N. (Year). “Transductive learning for zero-shot object detection”, in Liu, Y., Lee, J., Park, M., Kim, S., Yang, E., Hwang, S.J., et al. (2018). Learning to propagate labels: Transductive propagation network for few-shot learning. Mirza, M., and Osindero, S. (2014). Conditional generative adversarial nets. Hughes, Schmitt, Mou, Wang, Zhu (b0055) 2018; 15 Mozafari, M., Reddy, L., and VanRullen, R. (2020). Reconstructing Natural Scenes from fMRI Patterns using BigBiGAN. Yang, Zhang, Xiang, Torr, Hospedales (b0205) 2018 Wang (10.1016/j.bspc.2021.103397_b0195) 2020; 53 Koch (10.1016/j.bspc.2021.103397_b0065) 2015 Beliy (10.1016/j.bspc.2021.103397_b0010) 2019 Norman (10.1016/j.bspc.2021.103397_b0125) 2006; 10 Russakovsky (10.1016/j.bspc.2021.103397_b0140) 2015; 115 Styves (10.1016/j.bspc.2021.103397_b0180) 2017 Naselaris (10.1016/j.bspc.2021.103397_b0110) 2011; 56 Chong (10.1016/j.bspc.2021.103397_b0020) 2020; 408 Horikawa (10.1016/j.bspc.2021.103397_b0050) 2017; 8 10.1016/j.bspc.2021.103397_b0080 Guclu (10.1016/j.bspc.2021.103397_b0035) 2015; 35 LeCun (10.1016/j.bspc.2021.103397_b0075) 2015; 521 10.1016/j.bspc.2021.103397_b0130 Nishimoto (10.1016/j.bspc.2021.103397_b0120) 2011; 21 Wen (10.1016/j.bspc.2021.103397_b0200) 2017 Arjovsky (10.1016/j.bspc.2021.103397_b0005) 2017 Krizhevsky (10.1016/j.bspc.2021.103397_b0070) 2012 10.1016/j.bspc.2021.103397_b0095 Miyawaki (10.1016/j.bspc.2021.103397_b0100) 2008; 60 Chao (10.1016/j.bspc.2021.103397_b0015) 2016 10.1016/j.bspc.2021.103397_b0215 Shen (10.1016/j.bspc.2021.103397_b0155) 2019; 15 10.1016/j.bspc.2021.103397_b0210 Han (10.1016/j.bspc.2021.103397_b0045) 2019; 198 10.1016/j.bspc.2021.103397_b0135 Eickenberg (10.1016/j.bspc.2021.103397_b0025) 2016; 152 van Gerven (10.1016/j.bspc.2021.103397_b0185) 2010; 22 Seeliger (10.1016/j.bspc.2021.103397_b0145) 2018; 181 St-Yves (10.1016/j.bspc.2021.103397_b0175) 2018 Goodfellow (10.1016/j.bspc.2021.103397_b0030) 2014 10.1016/j.bspc.2021.103397_b0190 10.1016/j.bspc.2021.103397_b0090 Song (10.1016/j.bspc.2021.103397_b0170) 2018 Naselaris (10.1016/j.bspc.2021.103397_b0115) 2009; 63 10.1016/j.bspc.2021.103397_b0085 10.1016/j.bspc.2021.103397_b0165 Shi (10.1016/j.bspc.2021.103397_b0160) 2018 Güçlütürk (10.1016/j.bspc.2021.103397_b0040) 2017 10.1016/j.bspc.2021.103397_b0105 Shen (10.1016/j.bspc.2021.103397_b0150) 2018 Hughes (10.1016/j.bspc.2021.103397_b0055) 2018; 15 Yang (10.1016/j.bspc.2021.103397_b0205) 2018 Jiang (10.1016/j.bspc.2021.103397_b0060) 2019; 9 |
| References_xml | – volume: 181 start-page: 775 year: 2018 end-page: 785 ident: b0145 article-title: Generative adversarial networks for reconstructing natural images from brain activity publication-title: NeuroImage – volume: 408 start-page: 216 year: 2020 end-page: 230 ident: b0020 article-title: Graph-based semi-supervised learning: A review publication-title: Neurocomputing – year: 2015 ident: b0065 article-title: Siamese neural networks for one-shot image recognition publication-title: in: – year: 2018 ident: b0175 article-title: Generative adversarial networks conditioned on brain activity reconstruct seen images. publication-title: bioRxiv – volume: 63 start-page: 902 year: 2009 end-page: 915 ident: b0115 article-title: Bayesian reconstruction of natural images from human brain activity: neuron publication-title: Neuron – start-page: 1 year: 2017 end-page: 25 ident: b0200 article-title: Neural encoding and decoding with deep learning for dynamic natural vision publication-title: Cereb. Cortex – start-page: 1097 year: 2012 end-page: 1105 ident: b0070 article-title: ImageNet classification with deep convolutional neural networks publication-title: in: – volume: 115 start-page: 211 year: 2015 end-page: 252 ident: b0140 article-title: Imagenet large scale visual recognition challenge publication-title: Int. J. Comput. Vision – reference: Ren, Z., Li, J., Xue, X., Li, X., Yang, F., Jiao, Z., et al. (2019). Reconstructing Perceived Images from Brain Activity by Visually-guided Cognitive Representation and Adversarial Learning. – year: 2018 ident: b0150 article-title: End-to-end deep image reconstruction from human brain activity. publication-title: bioRxiv – reference: Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. – volume: 521 start-page: 436 year: 2015 end-page: 444 ident: b0075 article-title: Deep learning publication-title: Deep learning. – reference: 12. – reference: Zhang, C., Qiao, K., Wang, L., Tong, L., Zeng, Y., and Yan, B. (2018). Constraint-free natural image reconstruction from fMRI signals based on convolutional neural network. – volume: 53 start-page: 1 year: 2020 end-page: 34 ident: b0195 article-title: Generalizing from a few examples: A survey on few-shot learning publication-title: ACM Computing Surveys (CSUR) – reference: Mozafari, M., Reddy, L., and VanRullen, R. (2020). Reconstructing Natural Scenes from fMRI Patterns using BigBiGAN. – volume: 8 start-page: 15037 year: 2017 ident: b0050 article-title: Generic decoding of seen and imagined objects using hierarchical visual features publication-title: Nat. Commun. – start-page: 2672 year: 2014 end-page: 2680 ident: b0030 article-title: Generative adversarial nets publication-title: Advances in neural information processing systems – volume: 56 start-page: 400 year: 2011 end-page: 410 ident: b0110 article-title: Encoding and decoding in fMRI publication-title: Neuroimage – reference: Liu, Y., Lee, J., Park, M., Kim, S., Yang, E., Hwang, S.J., et al. (2018). Learning to propagate labels: Transductive propagation network for few-shot learning. – start-page: 6514 year: 2019 end-page: 6524 ident: b0010 article-title: From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI publication-title: Advances in Neural Information Processing Systems – volume: 60 start-page: 915 year: 2008 end-page: 929 ident: b0100 article-title: Visual image reconstruction from human brain activity using a combination of multiscale local image decoders publication-title: Neuron – reference: ), 6082-6091. – reference: , 6949-6958. – start-page: 4246 year: 2017 end-page: 4257 ident: b0040 article-title: Reconstructing perceived faces from brain activations with deep adversarial neural decoding publication-title: Adv. Neural Information Processing Systems – start-page: 52 year: 2016 end-page: 68 ident: b0015 article-title: An empirical study and analysis of generalized zero-shot learning for object recognition in the wild publication-title: : Springer – reference: , 6707-6717. – year: 2017 ident: b0180 article-title: The feature-weighted receptive field: an interpretable encoding model for complex feature spaces publication-title: Neuroimage – volume: 15 start-page: e1006633 year: 2019 ident: b0155 article-title: Deep image reconstruction from human brain activity publication-title: PLoS Comput. Biol. – start-page: 299 year: 2018 end-page: 315 ident: b0160 article-title: Transductive semi-supervised deep learning using min-max features publication-title: in: – volume: 15 start-page: 784 year: 2018 end-page: 788 ident: b0055 article-title: Identifying corresponding patches in SAR and optical images with a pseudo-siamese CNN publication-title: IEEE Geosci. Remote Sens. Lett. – reference: Misra, I., and Maaten, L.v.d. (2020). “Self-supervised learning of pretext-invariant representations”, in: – reference: Mescheder, L., Geiger, A., and Nowozin, S. (2018). Which training methods for GANs do actually converge? – reference: . – volume: 10 start-page: 424 year: 2006 end-page: 430 ident: b0125 article-title: Beyond mind-reading: multi-voxel pattern analysis of fMRI data publication-title: Trends in cognitive sciences – reference: Zhang, Y., Wu, Z., Peng, H., and Lin, S. (2020). “A transductive approach for video object segmentation”, in: – volume: 21 start-page: 1641 year: 2011 end-page: 1646 ident: b0120 article-title: Reconstructing visual experiences from brain activity evoked by natural movies publication-title: Curr. Biol. – reference: Rahman, S., Khan, S., and Barnes, N. (Year). “Transductive learning for zero-shot object detection”, in: – volume: 9 start-page: 4749 year: 2019 ident: b0060 article-title: Siamese reconstruction network: accurate image reconstruction from human brain activity by learning to compare publication-title: Appl. Sci. – volume: 198 start-page: 125 year: 2019 end-page: 136 ident: b0045 article-title: Variational autoencoder: An unsupervised model for encoding and decoding fMRI activity in visual cortex publication-title: NeuroImage – reference: Mirza, M., and Osindero, S. (2014). Conditional generative adversarial nets. – start-page: 1024 year: 2018 end-page: 1033 ident: b0170 article-title: Transductive unbiased embedding for zero-shot learning publication-title: in: – reference: VanRullen, R., and Reddy, L. (2018). Reconstructing Faces from fMRI Patterns using Deep Generative Neural Networks. – start-page: 214 year: 2017 end-page: 223 ident: b0005 article-title: Wasserstein generative adversarial networks publication-title: International Conference on Machine Learning – volume: 152 year: 2016 ident: b0025 article-title: Seeing it all: Convolutional network layers map the function of the human visual system publication-title: Neuroimage – volume: 22 start-page: 3127 year: 2010 end-page: 3142 ident: b0185 article-title: Neural decoding with hierarchical generative models publication-title: Neural Comput. – year: 2018 ident: b0205 article-title: Learning to compare – volume: 35 start-page: 10005 year: 2015 end-page: 10014 ident: b0035 article-title: Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream publication-title: J. Neurosci. – ident: 10.1016/j.bspc.2021.103397_b0130 – volume: 15 start-page: e1006633 issue: 1 year: 2019 ident: 10.1016/j.bspc.2021.103397_b0155 article-title: Deep image reconstruction from human brain activity publication-title: PLoS Comput. Biol. doi: 10.1371/journal.pcbi.1006633 – ident: 10.1016/j.bspc.2021.103397_b0215 doi: 10.1109/CVPR42600.2020.00698 – ident: 10.1016/j.bspc.2021.103397_b0090 – volume: 115 start-page: 211 issue: 3 year: 2015 ident: 10.1016/j.bspc.2021.103397_b0140 article-title: Imagenet large scale visual recognition challenge publication-title: Int. J. Comput. Vision doi: 10.1007/s11263-015-0816-y – volume: 22 start-page: 3127 issue: 12 year: 2010 ident: 10.1016/j.bspc.2021.103397_b0185 article-title: Neural decoding with hierarchical generative models publication-title: Neural Comput. doi: 10.1162/NECO_a_00047 – volume: 9 start-page: 4749 issue: 22 year: 2019 ident: 10.1016/j.bspc.2021.103397_b0060 article-title: Siamese reconstruction network: accurate image reconstruction from human brain activity by learning to compare publication-title: Appl. Sci. doi: 10.3390/app9224749 – volume: 408 start-page: 216 year: 2020 ident: 10.1016/j.bspc.2021.103397_b0020 article-title: Graph-based semi-supervised learning: A review publication-title: Neurocomputing doi: 10.1016/j.neucom.2019.12.130 – volume: 181 start-page: 775 year: 2018 ident: 10.1016/j.bspc.2021.103397_b0145 article-title: Generative adversarial networks for reconstructing natural images from brain activity publication-title: NeuroImage doi: 10.1016/j.neuroimage.2018.07.043 – volume: 60 start-page: 915 issue: 5 year: 2008 ident: 10.1016/j.bspc.2021.103397_b0100 article-title: Visual image reconstruction from human brain activity using a combination of multiscale local image decoders publication-title: Neuron doi: 10.1016/j.neuron.2008.11.004 – ident: 10.1016/j.bspc.2021.103397_b0165 – year: 2017 ident: 10.1016/j.bspc.2021.103397_b0180 article-title: The feature-weighted receptive field: an interpretable encoding model for complex feature spaces publication-title: Neuroimage – start-page: 6514 year: 2019 ident: 10.1016/j.bspc.2021.103397_b0010 article-title: From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI publication-title: Advances in Neural Information Processing Systems – ident: 10.1016/j.bspc.2021.103397_b0190 doi: 10.32470/CCN.2018.1031-0 – start-page: 214 year: 2017 ident: 10.1016/j.bspc.2021.103397_b0005 article-title: Wasserstein generative adversarial networks publication-title: International Conference on Machine Learning – ident: 10.1016/j.bspc.2021.103397_b0085 – volume: 521 start-page: 436 issue: 7553 year: 2015 ident: 10.1016/j.bspc.2021.103397_b0075 article-title: Deep learning publication-title: Deep learning. nature – year: 2018 ident: 10.1016/j.bspc.2021.103397_b0205 – start-page: 1024 year: 2018 ident: 10.1016/j.bspc.2021.103397_b0170 article-title: Transductive unbiased embedding for zero-shot learning – volume: 21 start-page: 1641 issue: 19 year: 2011 ident: 10.1016/j.bspc.2021.103397_b0120 article-title: Reconstructing visual experiences from brain activity evoked by natural movies publication-title: Curr. Biol. doi: 10.1016/j.cub.2011.08.031 – volume: 35 start-page: 10005 issue: 27 year: 2015 ident: 10.1016/j.bspc.2021.103397_b0035 article-title: Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream publication-title: J. Neurosci. doi: 10.1523/JNEUROSCI.5023-14.2015 – volume: 15 start-page: 784 issue: 5 year: 2018 ident: 10.1016/j.bspc.2021.103397_b0055 article-title: Identifying corresponding patches in SAR and optical images with a pseudo-siamese CNN publication-title: IEEE Geosci. Remote Sens. Lett. doi: 10.1109/LGRS.2018.2799232 – start-page: 2672 year: 2014 ident: 10.1016/j.bspc.2021.103397_b0030 article-title: Generative adversarial nets publication-title: Advances in neural information processing systems – year: 2018 ident: 10.1016/j.bspc.2021.103397_b0175 article-title: Generative adversarial networks conditioned on brain activity reconstruct seen images. publication-title: bioRxiv – volume: 56 start-page: 400 issue: 2 year: 2011 ident: 10.1016/j.bspc.2021.103397_b0110 article-title: Encoding and decoding in fMRI publication-title: Neuroimage doi: 10.1016/j.neuroimage.2010.07.073 – start-page: 4246 year: 2017 ident: 10.1016/j.bspc.2021.103397_b0040 article-title: Reconstructing perceived faces from brain activations with deep adversarial neural decoding publication-title: Adv. Neural Information Processing Systems – start-page: 1 year: 2017 ident: 10.1016/j.bspc.2021.103397_b0200 article-title: Neural encoding and decoding with deep learning for dynamic natural vision publication-title: Cereb. Cortex – year: 2015 ident: 10.1016/j.bspc.2021.103397_b0065 article-title: Siamese neural networks for one-shot image recognition – ident: 10.1016/j.bspc.2021.103397_b0105 doi: 10.1109/IJCNN48605.2020.9206960 – volume: 63 start-page: 902 issue: 6 year: 2009 ident: 10.1016/j.bspc.2021.103397_b0115 article-title: Bayesian reconstruction of natural images from human brain activity: neuron publication-title: Neuron doi: 10.1016/j.neuron.2009.09.006 – start-page: 299 year: 2018 ident: 10.1016/j.bspc.2021.103397_b0160 article-title: Transductive semi-supervised deep learning using min-max features – ident: 10.1016/j.bspc.2021.103397_b0135 – volume: 8 start-page: 15037 year: 2017 ident: 10.1016/j.bspc.2021.103397_b0050 article-title: Generic decoding of seen and imagined objects using hierarchical visual features publication-title: Nat. Commun. doi: 10.1038/ncomms15037 – volume: 198 start-page: 125 year: 2019 ident: 10.1016/j.bspc.2021.103397_b0045 article-title: Variational autoencoder: An unsupervised model for encoding and decoding fMRI activity in visual cortex publication-title: NeuroImage doi: 10.1016/j.neuroimage.2019.05.039 – start-page: 1097 year: 2012 ident: 10.1016/j.bspc.2021.103397_b0070 article-title: ImageNet classification with deep convolutional neural networks – ident: 10.1016/j.bspc.2021.103397_b0080 – volume: 152 year: 2016 ident: 10.1016/j.bspc.2021.103397_b0025 article-title: Seeing it all: Convolutional network layers map the function of the human visual system publication-title: Neuroimage – start-page: 52 year: 2016 ident: 10.1016/j.bspc.2021.103397_b0015 article-title: An empirical study and analysis of generalized zero-shot learning for object recognition in the wild publication-title: European conference on computer vision: Springer – volume: 53 start-page: 1 issue: 3 year: 2020 ident: 10.1016/j.bspc.2021.103397_b0195 article-title: Generalizing from a few examples: A survey on few-shot learning publication-title: ACM Computing Surveys (CSUR) doi: 10.1145/3386252 – year: 2018 ident: 10.1016/j.bspc.2021.103397_b0150 article-title: End-to-end deep image reconstruction from human brain activity. publication-title: bioRxiv – ident: 10.1016/j.bspc.2021.103397_b0210 doi: 10.3389/fnhum.2018.00242 – ident: 10.1016/j.bspc.2021.103397_b0095 doi: 10.1109/CVPR42600.2020.00674 – volume: 10 start-page: 424 issue: 9 year: 2006 ident: 10.1016/j.bspc.2021.103397_b0125 article-title: Beyond mind-reading: multi-voxel pattern analysis of fMRI data publication-title: Trends in cognitive sciences doi: 10.1016/j.tics.2006.07.005 |
| SSID | ssj0048714 |
| Score | 2.2853737 |
| Snippet | •We proposed the mutual promotion of visual encoding and reconstruction models.•We designed alternating optimization bassed on seim-supervised learning.•We... |
| SourceID | crossref elsevier |
| SourceType | Enrichment Source Index Database Publisher |
| StartPage | 103397 |
| SubjectTerms | Alternating optimization fMRI Shared autoencoder Visual encoding Visual reconstruction |
| Title | Reconstructing natural images from human fMRI by alternating encoding and decoding with shared autoencoder regularization |
| URI | https://dx.doi.org/10.1016/j.bspc.2021.103397 |
| Volume | 73 |
| WOSCitedRecordID | wos000782654300004&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVESC databaseName: Elsevier SD Freedom Collection Journals 2021 customDbUrl: eissn: 1746-8108 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0048714 issn: 1746-8094 databaseCode: AIEXJ dateStart: 20060101 isFulltext: true titleUrlDefault: https://www.sciencedirect.com providerName: Elsevier |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtV1LT9tAEF6loYdyqKAPQVuqPfQWGdne9euIEFWpCkIVVXOz7H2AEThRCIj8lf7azuzDcWiL4NCLlax211bmy3hmduYbQj6lAowKkfKApVwEXGodFDqvg0wBWmpkZ8lNofC37Pg4H4-Lk8Hgl6-Fub3M2ja_uyum_1XUMAbCxtLZJ4i72xQG4DMIHa4gdrg-SvDoUHpa2PZsZJg7kVnjqkI6B1NOYhvz6aPvh2h9mgNzDArCbKS1lL5uUSr3xeaxn5tc9epmPjGz1Gw0M43sZ66Uc-V82FT125LL5gwt3qmtSPB7uxT5Lu7aVBOb3NEsEw5c3UgPwD9ddBs86MXNcriLeu-fN_0wBnjAXR6X07wZR2Zk2_HYq-aM9XRrFDJmc3n_UPs2AnGxW19PkZYyjnaXk1c5tu-9-7qMRJ_sdlHiHiXuUdo9npG1OEuKfEjW9g4Pxl_9ex48PcMc3z24K8my2YP3n-TvZk_PlDndIC-dD0L3LHY2yUC1r8h6j5nyNVmsoog6FFGLIoooogZFFFFE6wXtoYh6FFGQNPUooogialFEeyiiqyh6Q358Pjjd_xK4Lh2BYGE4D5jOVMWLOtU8EaHkmQIPWBa8rkOuEylEovFwPJI8lnEiWSGTSEudyrioBExnb8mwnbRqi9BcREzFNQP9kcA-VR4xDQsltggArRFuk8j_iKVwFPbYSeWy_Lf4tsmoWzO1BC4Pzk68bEpnglrTsgSoPbDu3ZPu8p68WP4FPpAhCFPtkOfidt5czz46nP0GEgetFg |
| linkProvider | Elsevier |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Reconstructing+natural+images+from+human+fMRI+by+alternating+encoding+and+decoding+with+shared+autoencoder+regularization&rft.jtitle=Biomedical+signal+processing+and+control&rft.au=Qiao%2C+Kai&rft.au=Chen%2C+Jian&rft.au=Wang%2C+Linyuan&rft.au=Zhang%2C+Chi&rft.date=2022-03-01&rft.issn=1746-8094&rft.volume=73&rft.spage=103397&rft_id=info:doi/10.1016%2Fj.bspc.2021.103397&rft.externalDBID=n%2Fa&rft.externalDocID=10_1016_j_bspc_2021_103397 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1746-8094&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1746-8094&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1746-8094&client=summon |