Zobrazit v EDS

Reconstruction of natural images from human fMRI using a three-stage multi-level deep fusion model.

Uloženo v:
Podrobná bibliografie
Název: Reconstruction of natural images from human fMRI using a three-stage multi-level deep fusion model.
Autoři: Meng L; School of Information Science and Engineering, Northeastern University, Shenyang 110819, China. Electronic address: menglu@mail.neu.edu.cn., Tang Z; School of Information Science and Engineering, Northeastern University, Shenyang 110819, China., Liu Y; Peng Cheng Laboratory, China.
Zdroj: Journal of neuroscience methods [J Neurosci Methods] 2024 Nov; Vol. 411, pp. 110269. Date of Electronic Publication: 2024 Aug 31.
Způsob vydávání: Journal Article
Jazyk: English
Informace o časopise: Publisher: Elsevier/North-Holland Biomedical Press Country of Publication: Netherlands NLM ID: 7905558 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1872-678X (Electronic) Linking ISSN: 01650270 NLM ISO Abbreviation: J Neurosci Methods Subsets: MEDLINE
Imprint Name(s): Original Publication: Amsterdam, Elsevier/North-Holland Biomedical Press.
Výrazy ze slovníku MeSH: Magnetic Resonance Imaging*/methods , Image Processing, Computer-Assisted*/methods , Brain*/diagnostic imaging , Brain*/physiology, Humans ; Brain Mapping/methods ; Deep Learning
Abstrakt: Background: Image reconstruction is a critical task in brain decoding research, primarily utilizing functional magnetic resonance imaging (fMRI) data. However, due to challenges such as limited samples in fMRI data, the quality of reconstruction results often remains poor.
New Method: We proposed a three-stage multi-level deep fusion model (TS-ML-DFM). The model employed a three-stage training process, encompassing components such as image encoders, generators, discriminators, and fMRI encoders. In this method, we incorporated distinct supplementary features derived separately from depth images and original images. Additionally, the method integrated several components, including a random shift module, dual attention module, and multi-level feature fusion module.
Results: In both qualitative and quantitative comparisons on the Horikawa17 and VanGerven10 datasets, our method exhibited excellent performance.
Comparison With Existing Methods: For example, on the primary Horikawa17 dataset, our method was compared with other leading methods based on metrics the average hash value, histogram similarity, mutual information, structural similarity accuracy, AlexNet(2), AlexNet(5), and pairwise human perceptual similarity accuracy. Compared to the second-ranked results in each metric, the proposed method achieved improvements of 0.99 %, 3.62 %, 3.73 %, 2.45 %, 3.51 %, 0.62 %, and 1.03 %, respectively. In terms of the SwAV top-level semantic metric, a substantial improvement of 10.53 % was achieved compared to the second-ranked result in the pixel-level reconstruction methods.
Conclusions: The TS-ML-DFM method proposed in this study, when applied to decoding brain visual patterns using fMRI data, has outperformed previous algorithms, thereby facilitating further advancements in research within this field.
(Copyright © 2024 The Authors. Published by Elsevier B.V. All rights reserved.)
Competing Interests: Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Contributed Indexing: Keywords: Depth image; FMRI; Self-supervised learning; Three-stage process; Visual stimulus decoding
Entry Date(s): Date Created: 20240902 Date Completed: 20240915 Latest Revision: 20240915
Update Code: 20250114
DOI: 10.1016/j.jneumeth.2024.110269
PMID: 39222796
Databáze: MEDLINE
Popis
Abstrakt:Background: Image reconstruction is a critical task in brain decoding research, primarily utilizing functional magnetic resonance imaging (fMRI) data. However, due to challenges such as limited samples in fMRI data, the quality of reconstruction results often remains poor.<br />New Method: We proposed a three-stage multi-level deep fusion model (TS-ML-DFM). The model employed a three-stage training process, encompassing components such as image encoders, generators, discriminators, and fMRI encoders. In this method, we incorporated distinct supplementary features derived separately from depth images and original images. Additionally, the method integrated several components, including a random shift module, dual attention module, and multi-level feature fusion module.<br />Results: In both qualitative and quantitative comparisons on the Horikawa17 and VanGerven10 datasets, our method exhibited excellent performance.<br />Comparison With Existing Methods: For example, on the primary Horikawa17 dataset, our method was compared with other leading methods based on metrics the average hash value, histogram similarity, mutual information, structural similarity accuracy, AlexNet(2), AlexNet(5), and pairwise human perceptual similarity accuracy. Compared to the second-ranked results in each metric, the proposed method achieved improvements of 0.99 %, 3.62 %, 3.73 %, 2.45 %, 3.51 %, 0.62 %, and 1.03 %, respectively. In terms of the SwAV top-level semantic metric, a substantial improvement of 10.53 % was achieved compared to the second-ranked result in the pixel-level reconstruction methods.<br />Conclusions: The TS-ML-DFM method proposed in this study, when applied to decoding brain visual patterns using fMRI data, has outperformed previous algorithms, thereby facilitating further advancements in research within this field.<br /> (Copyright © 2024 The Authors. Published by Elsevier B.V. All rights reserved.)
ISSN:1872-678X
DOI:10.1016/j.jneumeth.2024.110269