MSCSCformer: Multiscale Convolutional Sparse Coding-Based Transformer for Pansharpening

With the increasing significance of high-quality, high-resolution multispectral images (HRMSs) in various domains, pansharpening, which fuses low-resolution multispectral images (LRMSs) with high-resolution panchromatic (PAN) images, has gained considerable attention. However, current deep-learning...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on geoscience and remote sensing Vol. 62; pp. 1 - 12
Main Authors: Ye, Yongxu, Wang, Tingting, Fang, Faming, Zhang, Guixu
Format: Journal Article
Language:English
Published: New York IEEE 2024
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Ablation
Coding
Convolutional neural networks
Deep unfolding network
Feature extraction
High resolution
Image quality
Image resolution
Information processing
Iterative methods
multiscale convolution sparse coding (MS-CSC)
Optimization
Pansharpening
pansharpening (PAN)
remote sensing
Spectral bands
Task analysis
transformer
Transformers
ISSN:0196-2892, 1558-0644
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract With the increasing significance of high-quality, high-resolution multispectral images (HRMSs) in various domains, pansharpening, which fuses low-resolution multispectral images (LRMSs) with high-resolution panchromatic (PAN) images, has gained considerable attention. However, current deep-learning (DL) methods have limitations in capturing global long-range dependencies and incorporating spectral characteristics across different spectral bands of multispectral (MS) images. Additionally, model-based approaches do not effectively utilize the multiscale information between LRMS and HRMS data, limiting their further performance enhancement. To address these limitations, we propose a new observation model based on multiscale convolutional sparse coding (MS-CSC) and design a novel multiscale hybrid spatial-spectral transformer (MSHST) for the unfolding networks. The MS-CSC-based observation model aims to fuse multiscale information, while the MSHST incorporates spatial self-attention to capture global long-range dependencies and spectral self-attention to capture the interband correlation. Experimental results demonstrate the superiority of our method over other state-of-the-art approaches in both reduced-resolution and full-resolution evaluations. Ablation experiments further validate the effectiveness of the proposed multiscale model and MSHST. Code is available at https://github.com/Eternityyx/MSCSCformer .
AbstractList With the increasing significance of high-quality, high-resolution multispectral images (HRMSs) in various domains, pansharpening, which fuses low-resolution multispectral images (LRMSs) with high-resolution panchromatic (PAN) images, has gained considerable attention. However, current deep-learning (DL) methods have limitations in capturing global long-range dependencies and incorporating spectral characteristics across different spectral bands of multispectral (MS) images. Additionally, model-based approaches do not effectively utilize the multiscale information between LRMS and HRMS data, limiting their further performance enhancement. To address these limitations, we propose a new observation model based on multiscale convolutional sparse coding (MS-CSC) and design a novel multiscale hybrid spatial–spectral transformer (MSHST) for the unfolding networks. The MS-CSC-based observation model aims to fuse multiscale information, while the MSHST incorporates spatial self-attention to capture global long-range dependencies and spectral self-attention to capture the interband correlation. Experimental results demonstrate the superiority of our method over other state-of-the-art approaches in both reduced-resolution and full-resolution evaluations. Ablation experiments further validate the effectiveness of the proposed multiscale model and MSHST. Code is available at https://github.com/Eternityyx/MSCSCformer .
Author Wang, Tingting
Fang, Faming
Zhang, Guixu
Ye, Yongxu
Author_xml – sequence: 1
  givenname: Yongxu
  orcidid: 0000-0003-1914-9780
  surname: Ye
  fullname: Ye, Yongxu
  email: 51215901060@stu.ecnu.edu
  organization: School of Computer Science and Technology, East China Normal University, Shanghai, China
– sequence: 2
  givenname: Tingting
  orcidid: 0009-0008-8433-8063
  surname: Wang
  fullname: Wang, Tingting
  email: tingtingwang@cs.ecnu.edu.cn
  organization: School of Computer Science and Technology, East China Normal University, Shanghai, China
– sequence: 3
  givenname: Faming
  orcidid: 0000-0003-4511-4813
  surname: Fang
  fullname: Fang, Faming
  email: fmfang@cs.ecnu.edu.cn
  organization: School of Computer Science and Technology, East China Normal University, Shanghai, China
– sequence: 4
  givenname: Guixu
  orcidid: 0000-0003-4720-6607
  surname: Zhang
  fullname: Zhang, Guixu
  email: gxzhang@cs.ecnu.edu.cn
  organization: School of Computer Science and Technology, East China Normal University, Shanghai, China
BookMark eNp9kFFLwzAQx4NMcJt-AMGHgs-duaRJE9-06BQ2FDvxsSRtoh1dO5NW8Nvb0j2IDz4dd_x-x91_hiZ1UxuEzgEvALC82ixf0gXBJFpQKoEydoSmwJgIMY-iCZpikDwkQpITNPN-izFEDOIpelunSZrYxu2Muw7WXdWWPleVCZKm_mqqri2bWlVBulfOD8OirN_DW-VNEWycqv1oBn0Jnvv2Q7m9qXvmFB1bVXlzdqhz9Hp_t0kewtXT8jG5WYU5kVEbFlZwBlFOBbOs0MQqwbUEQgxogy3XlvICLOhYUi4V0FwrAC2tAKG1UHSOLse9e9d8dsa32bbpXH-yzyiOZCQkp7Kn4pHKXeO9MzbLy1YNv7VOlVUGOBtSzIYUsyHF7JBib8Ifc-_KnXLf_zoXo1MaY37xDPMYOP0BtpuAlw
CODEN IGRSD2
CitedBy_id crossref_primary_10_3390_e27060567
crossref_primary_10_1109_TGRS_2025_3580088
crossref_primary_10_1109_TGRS_2025_3585606
crossref_primary_10_1109_TGRS_2025_3599130
crossref_primary_10_1109_TGRS_2025_3544755
crossref_primary_10_1109_JSTARS_2025_3585751
Cites_doi 10.1016/j.inffus.2012.05.003
10.1109/TGRS.2021.3139190
10.1109/MGRS.2022.3187652
10.1080/01431161.2012.744882
10.1109/JSTARS.2022.3193182
10.1109/CVPR52688.2022.00183
10.1109/MGRS.2022.3170092
10.1109/TGRS.2020.3048257
10.1109/TGRS.2020.3015878
10.1109/CVPR46437.2021.00142
10.1109/TGRS.2022.3197438
10.1109/TIP.2013.2258355
10.1109/TCYB.2019.2910151
10.1109/TGRS.2022.3168465
10.1109/TPAMI.2023.3279050
10.1109/TGRS.2010.2067219
10.1109/TGRS.2020.2964627
10.1137/080716542
10.1016/j.inffus.2021.09.002
10.1109/ICCV.2015.69
10.1109/TGRS.2007.904923
10.1109/TNNLS.2021.3084745
10.1109/JSTARS.2014.2347072
10.1109/TGRS.2010.2051674
10.1109/IGARSS.2017.8128408
10.1109/TGRS.2019.2906073
10.1049/el:20080522
10.1109/LGRS.2007.909934
10.1109/TGRS.2022.3225563
10.1109/MGRS.2022.3145854
10.1109/LGRS.2013.2281996
10.1109/TGRS.2007.901007
10.1109/TGRS.2022.3152425
10.1109/TPAMI.2015.2439281
10.1109/TGRS.2022.3227405
10.1109/TGRS.2022.3163887
10.1109/tgrs.2021.3115501
10.3390/rs8070594
10.1109/TGRS.2020.3007884
10.1109/ICCV48922.2021.00986
10.1109/TIP.2003.819861
10.1109/CVPR.2018.00695
10.1016/j.inffus.2020.04.006
10.1109/TGRS.2014.2361734
10.1109/TIP.2018.2866954
10.1109/CVPR52688.2022.00564
10.1109/tnnls.2023.3313202
10.1109/CVPR52688.2022.00181
ContentType Journal Article
Copyright Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024
Copyright_xml – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2024
DBID 97E
RIA
RIE
AAYXX
CITATION
7UA
8FD
C1K
F1W
FR3
H8D
H96
KR7
L.G
L7M
DOI 10.1109/TGRS.2024.3391355
DatabaseName IEEE Xplore (IEEE)
IEEE All-Society Periodicals Package (ASPP) 1998–Present
IEEE Electronic Library (IEL)
CrossRef
Water Resources Abstracts
Technology Research Database
Environmental Sciences and Pollution Management
ASFA: Aquatic Sciences and Fisheries Abstracts
Engineering Research Database
Aerospace Database
Aquatic Science & Fisheries Abstracts (ASFA) 2: Ocean Technology, Policy & Non-Living Resources
Civil Engineering Abstracts
Aquatic Science & Fisheries Abstracts (ASFA) Professional
Advanced Technologies Database with Aerospace
DatabaseTitle CrossRef
Aerospace Database
Civil Engineering Abstracts
Aquatic Science & Fisheries Abstracts (ASFA) Professional
Aquatic Science & Fisheries Abstracts (ASFA) 2: Ocean Technology, Policy & Non-Living Resources
Technology Research Database
ASFA: Aquatic Sciences and Fisheries Abstracts
Engineering Research Database
Advanced Technologies Database with Aerospace
Water Resources Abstracts
Environmental Sciences and Pollution Management
DatabaseTitleList Aerospace Database
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Xplore
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Engineering
Physics
EISSN 1558-0644
EndPage 12
ExternalDocumentID 10_1109_TGRS_2024_3391355
10506716
Genre orig-research
GrantInformation_xml – fundername: National Key Research and Development Program of China
  grantid: 2022ZD0161800
  funderid: 10.13039/501100012166
– fundername: National Natural Science Foundation of China
  grantid: 62271203; 62202173
  funderid: 10.13039/501100001809
– fundername: Shanghai Rising-Star Program
  grantid: 21QA1402500
  funderid: 10.13039/501100013105
GroupedDBID -~X
0R~
29I
4.4
5GY
5VS
6IK
97E
AAJGR
AARMG
AASAJ
AAWTH
ABAZT
ABQJQ
ABVLG
ACGFO
ACGFS
ACIWK
ACNCT
AENEX
AETIX
AFRAH
AGQYO
AGSQL
AHBIQ
AI.
AIBXA
AKJIK
AKQYR
ALLEH
ALMA_UNASSIGNED_HOLDINGS
ASUFR
ATWAV
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CS3
DU5
EBS
EJD
F5P
HZ~
H~9
IBMZZ
ICLAB
IFIPE
IFJZH
IPLJI
JAVBF
LAI
M43
O9-
OCL
P2P
RIA
RIE
RNS
RXW
TAE
TN5
VH1
Y6R
AAYXX
CITATION
7UA
8FD
C1K
F1W
FR3
H8D
H96
KR7
L.G
L7M
ID FETCH-LOGICAL-c294t-df86514c385f5db2fa86b9122e1be0f6bf36d1f1b79369a13cba11b9f818bb8a3
IEDL.DBID RIE
ISICitedReferencesCount 11
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001214652600018&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN 0196-2892
IngestDate Mon Jun 30 08:22:19 EDT 2025
Sat Nov 29 03:32:35 EST 2025
Tue Nov 18 22:28:56 EST 2025
Wed Aug 27 02:06:29 EDT 2025
IsPeerReviewed true
IsScholarly true
Language English
License https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html
https://doi.org/10.15223/policy-029
https://doi.org/10.15223/policy-037
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c294t-df86514c385f5db2fa86b9122e1be0f6bf36d1f1b79369a13cba11b9f818bb8a3
Notes ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ORCID 0000-0003-1914-9780
0000-0003-4511-4813
0009-0008-8433-8063
0000-0003-4720-6607
PQID 3049489639
PQPubID 85465
PageCount 12
ParticipantIDs ieee_primary_10506716
crossref_citationtrail_10_1109_TGRS_2024_3391355
crossref_primary_10_1109_TGRS_2024_3391355
proquest_journals_3049489639
PublicationCentury 2000
PublicationDate 20240000
2024-00-00
20240101
PublicationDateYYYYMMDD 2024-01-01
PublicationDate_xml – year: 2024
  text: 20240000
PublicationDecade 2020
PublicationPlace New York
PublicationPlace_xml – name: New York
PublicationTitle IEEE transactions on geoscience and remote sensing
PublicationTitleAbbrev TGRS
PublicationYear 2024
Publisher IEEE
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml – name: IEEE
– name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References ref13
ref12
ref15
ref14
ref52
ref11
ref17
ref16
ref19
ref18
Yuhas (ref47); 1
Wald (ref43) 1997; 63
Schowengerdt (ref10) 2006
ref51
ref46
ref45
ref48
ref42
ref41
ref44
ref49
ref8
ref7
ref9
ref4
ref3
ref6
ref5
ref40
ref35
ref34
ref37
ref36
ref31
ref30
ref33
ref32
Dosovitskiy (ref50)
ref2
ref1
ref39
ref38
ref24
ref23
ref26
ref25
ref20
ref22
ref21
ref28
ref27
ref29
References_xml – ident: ref3
  doi: 10.1016/j.inffus.2012.05.003
– ident: ref39
  doi: 10.1109/TGRS.2021.3139190
– ident: ref44
  doi: 10.1109/MGRS.2022.3187652
– ident: ref14
  doi: 10.1080/01431161.2012.744882
– ident: ref18
  doi: 10.1109/JSTARS.2022.3193182
– ident: ref42
  doi: 10.1109/CVPR52688.2022.00183
– ident: ref49
  doi: 10.1109/MGRS.2022.3170092
– ident: ref20
  doi: 10.1109/TGRS.2020.3048257
– ident: ref23
  doi: 10.1109/TGRS.2020.3015878
– ident: ref35
  doi: 10.1109/CVPR46437.2021.00142
– ident: ref36
  doi: 10.1109/TGRS.2022.3197438
– ident: ref13
  doi: 10.1109/TIP.2013.2258355
– ident: ref15
  doi: 10.1109/TCYB.2019.2910151
– ident: ref28
  doi: 10.1109/TGRS.2022.3168465
– ident: ref34
  doi: 10.1109/TPAMI.2023.3279050
– ident: ref16
  doi: 10.1109/TGRS.2010.2067219
– ident: ref2
  doi: 10.1109/TGRS.2020.2964627
– ident: ref41
  doi: 10.1137/080716542
– ident: ref25
  doi: 10.1016/j.inffus.2021.09.002
– ident: ref19
  doi: 10.1109/ICCV.2015.69
– ident: ref48
  doi: 10.1109/TGRS.2007.904923
– ident: ref31
  doi: 10.1109/TNNLS.2021.3084745
– ident: ref17
  doi: 10.1109/JSTARS.2014.2347072
– ident: ref6
  doi: 10.1109/TGRS.2010.2051674
– ident: ref4
  doi: 10.1109/IGARSS.2017.8128408
– ident: ref8
  doi: 10.1109/TGRS.2019.2906073
– volume-title: Remote Sensing: Models and Methods for Image Processing
  year: 2006
  ident: ref10
– ident: ref45
  doi: 10.1049/el:20080522
– ident: ref9
  doi: 10.1109/LGRS.2007.909934
– ident: ref38
  doi: 10.1109/TGRS.2022.3225563
– ident: ref1
  doi: 10.1109/MGRS.2022.3145854
– ident: ref11
  doi: 10.1109/LGRS.2013.2281996
– ident: ref7
  doi: 10.1109/TGRS.2007.901007
– ident: ref29
  doi: 10.1109/TGRS.2022.3152425
– ident: ref24
  doi: 10.1109/TPAMI.2015.2439281
– ident: ref30
  doi: 10.1109/TGRS.2022.3227405
– ident: ref33
  doi: 10.1109/TGRS.2022.3163887
– ident: ref37
  doi: 10.1109/tgrs.2021.3115501
– ident: ref21
  doi: 10.3390/rs8070594
– ident: ref22
  doi: 10.1109/TGRS.2020.3007884
– ident: ref52
  doi: 10.1109/ICCV48922.2021.00986
– ident: ref46
  doi: 10.1109/TIP.2003.819861
– volume: 1
  start-page: 1
  volume-title: Proc. JPL, Summaries 3rd Annu. JPL Airborne Geosci. Workshop. AVIRIS Workshop
  ident: ref47
  article-title: Discrimination among semi-arid landscape endmembers using the spectral angle mapper (SAM) algorithm
– ident: ref40
  doi: 10.1109/CVPR.2018.00695
– ident: ref32
  doi: 10.1016/j.inffus.2020.04.006
– start-page: 1
  volume-title: Proc. ICLR
  ident: ref50
  article-title: An image is worth 16×16 words: Transformers for image recognition at scale
– ident: ref5
  doi: 10.1109/TGRS.2014.2361734
– ident: ref12
  doi: 10.1109/TIP.2018.2866954
– volume: 63
  start-page: 691
  issue: 6
  year: 1997
  ident: ref43
  article-title: Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images
  publication-title: Photogramm. Eng. Remote Sens.
– ident: ref51
  doi: 10.1109/CVPR52688.2022.00564
– ident: ref27
  doi: 10.1109/tnnls.2023.3313202
– ident: ref26
  doi: 10.1109/CVPR52688.2022.00181
SSID ssj0014517
Score 2.4924226
Snippet With the increasing significance of high-quality, high-resolution multispectral images (HRMSs) in various domains, pansharpening, which fuses low-resolution...
SourceID proquest
crossref
ieee
SourceType Aggregation Database
Enrichment Source
Index Database
Publisher
StartPage 1
SubjectTerms Ablation
Coding
Convolutional neural networks
Deep unfolding network
Feature extraction
High resolution
Image quality
Image resolution
Information processing
Iterative methods
multiscale convolution sparse coding (MS-CSC)
Optimization
Pansharpening
pansharpening (PAN)
remote sensing
Spectral bands
Task analysis
transformer
Transformers
Title MSCSCformer: Multiscale Convolutional Sparse Coding-Based Transformer for Pansharpening
URI https://ieeexplore.ieee.org/document/10506716
https://www.proquest.com/docview/3049489639
Volume 62
WOSCitedRecordID wos001214652600018&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
journalDatabaseRights – providerCode: PRVIEE
  databaseName: IEEE Xplore
  customDbUrl:
  eissn: 1558-0644
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0014517
  issn: 0196-2892
  databaseCode: RIE
  dateStart: 19800101
  isFulltext: true
  titleUrlDefault: https://ieeexplore.ieee.org/
  providerName: IEEE
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEB5sUdCDz4rVKnvwJEQ3m93sxpsWqxdFbEVvy-aFB2nLVvv7nWRTKYiCp32QhJAvyTeTycwAnBa0EiKVKbFccpIqS0mBvEi0jnVONbOKaZ9sIn94KF5fxWNwVve-MMYYf_nMnLtXb8vXE_XpjspwhWe4uVLeglae88ZZ69tkkGY0-EZzglpEEkyYNBYXo9unIaqCSXrOmKDMufUtkZDPqvJjK_b8Mtj6Z8-2YTMIktFVg_wOrJjxLmwshRfchTV_vVPN9uDlftgf9p18aurLyDvdzhAcE_Un43mYfNjYcIpqrvvpCI1cI8HpaLSQbE0d4SN6xM83Z7RxByodeB7cjPp3JKRUICoR6QfRtuAoIilWZDbTMrFVwaWgSWKoNDHiZRnX1FKZu0R_FWVKVpRKYZHXpSwqtg_t8WRsDiCiDKlNK0W1kqnkHFdynGHNKk91lSnbhXgxxqUK8cZd2ov30usdsSgdLKWDpQywdOHsu8q0CbbxV-GOw2GpYANBF3oLJMuwHmcl82FwsIvi8JdqR7DuWm9OV3rQ_qg_zTGsqjkCUp_4qfYFZDXRcw
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEB58oh58i_W5B09CdLPJphtvWnyhFrEVvS2bFx6klVb9_U6yqRREwdM-SDZhvyTfTCYzA3BQ0EpKrjhxQgnCtaOkQF4kxqSmSQ1zmpmQbKLZbhfPz_I-OqsHXxhrbTh8Zo_8bbDlm77-8FtlOMNzXFypmITpnPMsrd21vo0GPKfRO1oQ1COyaMSkqTzuXj50UBnM-BFjkjLv2DdGQyGvyo_FODDMxdI_-7YMi1GUTE5r7FdgwvZWYWEswOAqzIYDnnq4Bk93nVan5SVUOzhJgtvtEOGxSavf-4zDDz_WeUNF17_0lEbOkOJM0h3JtnaQ4CW5x8cXb7bxWyrr8Hhx3m1dkZhUgehM8ndiXCFQSNKsyF1uVOaqQihJs8xSZVNEzDFhqKOq6VP9VZRpVVGqpENmV6qo2AZM9fo9uwkJZUhuRmtqtOJKCJzLaY41qyY3Va5dA9LRPy51jDjuE1-8lkHzSGXpYSk9LGWEpQGH31Xe6nAbfxVe9ziMFawhaMDOCMkyzshhyUIgHOyi3Pql2j7MXXXvbsvb6_bNNsz7luq9lh2Yeh982F2Y0Z8IzmAvDLsvclPUug
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=MSCSCformer%3A+Multiscale+Convolutional+Sparse+Coding-Based+Transformer+for+Pansharpening&rft.jtitle=IEEE+transactions+on+geoscience+and+remote+sensing&rft.au=Ye%2C+Yongxu&rft.au=Wang%2C+Tingting&rft.au=Fang%2C+Faming&rft.au=Zhang%2C+Guixu&rft.date=2024&rft.pub=IEEE&rft.issn=0196-2892&rft.volume=62&rft.spage=1&rft.epage=12&rft_id=info:doi/10.1109%2FTGRS.2024.3391355&rft.externalDocID=10506716
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0196-2892&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0196-2892&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0196-2892&client=summon