Fusing Transformers in a Tuning Fork Structure for Hyperspectral Image Classification Across Disjoint Samples
Uložené v:
| Názov: | Fusing Transformers in a Tuning Fork Structure for Hyperspectral Image Classification Across Disjoint Samples |
|---|---|
| Autori: | Muhammad Ahmad, Muhammad Usama, Manuel Mazzara, Salvatore Distefano, Hamad Ahmed Altuwaijri, Silvia Liberata Ullo |
| Zdroj: | IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol 17, Pp 18167-18181 (2024) |
| Informácie o vydavateľovi: | Institute of Electrical and Electronics Engineers (IEEE), 2024. |
| Rok vydania: | 2024 |
| Predmety: | hyperspectral image classification (HSIC), Ocean engineering, QC801-809, Geophysics. Cosmic physics, 3-D Swin Transformer (3DST), feature fusion, spatial–spectral transformer (SST), spatial–spectral features, TC1501-1800 |
| Popis: | The 3-D swin transformer (3DST) and spatial–spectral transformer (SST) each excel in capturing distinct aspects of image information: the 3DST with hierarchical attention and window-based processing, and the SST with self-attention mechanisms for long-range dependencies. However, applying them independently reveals the following limitations: the 3DST struggles with spectral information, while the SST lacks in capturing fine spatial details. In this article, we propose a novel tuning fork fusion approach to overcome these shortcomings, integrating the 3DST and SST to enhance the hyperspectral image (HSI) classification (HSIC). Our method integrates the hierarchical attention mechanism from the 3DST with the long-range dependence modeling of the SST. This combination refines spatial and spectral information representation and merges insights from both transformers at a fine-grained level. By emphasizing the fusion of attention mechanisms from both architectures, our approach significantly enhances the model's ability to capture complex spatial–spectral relationships, resulting in improved HSIC accuracy. In addition, we highlight the importance of disjoint training, validation, and test samples to enhance model generalization. Experimentation on benchmark HSI datasets demonstrates the superiority of our fusion approach over other state-of-the-art methods and standalone transformers. The source code has been developed from scratch and will be made public upon acceptance. |
| Druh dokumentu: | Article |
| ISSN: | 2151-1535 1939-1404 |
| DOI: | 10.1109/jstars.2024.3465831 |
| Prístupová URL adresa: | https://doaj.org/article/db3482e359d246c78013a2c7bebf6d7a |
| Rights: | CC BY NC ND |
| Prístupové číslo: | edsair.doi.dedup.....1b1218f96565a722766ee5c44d6a706f |
| Databáza: | OpenAIRE |
Full Text Finder 
Nájsť tento článok vo Web of Science | Abstrakt: | The 3-D swin transformer (3DST) and spatial–spectral transformer (SST) each excel in capturing distinct aspects of image information: the 3DST with hierarchical attention and window-based processing, and the SST with self-attention mechanisms for long-range dependencies. However, applying them independently reveals the following limitations: the 3DST struggles with spectral information, while the SST lacks in capturing fine spatial details. In this article, we propose a novel tuning fork fusion approach to overcome these shortcomings, integrating the 3DST and SST to enhance the hyperspectral image (HSI) classification (HSIC). Our method integrates the hierarchical attention mechanism from the 3DST with the long-range dependence modeling of the SST. This combination refines spatial and spectral information representation and merges insights from both transformers at a fine-grained level. By emphasizing the fusion of attention mechanisms from both architectures, our approach significantly enhances the model's ability to capture complex spatial–spectral relationships, resulting in improved HSIC accuracy. In addition, we highlight the importance of disjoint training, validation, and test samples to enhance model generalization. Experimentation on benchmark HSI datasets demonstrates the superiority of our fusion approach over other state-of-the-art methods and standalone transformers. The source code has been developed from scratch and will be made public upon acceptance. |
|---|---|
| ISSN: | 21511535 19391404 |
| DOI: | 10.1109/jstars.2024.3465831 |