A General Survey on Attention Mechanisms in Deep Learning

Attention is an important mechanism that can be employed for a variety of deep learning models across many different domains and tasks. This survey provides an overview of the most important attention mechanisms proposed in the literature. The various attention mechanisms are explained by means of a...

Ausführliche Beschreibung

Gespeichert in:

Bibliographische Detailangaben
Veröffentlicht in:	IEEE transactions on knowledge and data engineering Jg. 35; H. 4; S. 3279 - 3298
Hauptverfasser:	Brauwers, Gianni, Frasincar, Flavius
Format:	Journal Article
Sprache:	Englisch
Veröffentlicht:	New York IEEE 01.04.2023 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Schlagworte:	Attention models Biological system modeling Computational modeling Data models Deep learning Feature extraction introductory and survey neural nets supervised learning Task analysis Taxonomy
ISSN:	1041-4347, 1558-2191
Online-Zugang:	Volltext
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!

Abstract	Attention is an important mechanism that can be employed for a variety of deep learning models across many different domains and tasks. This survey provides an overview of the most important attention mechanisms proposed in the literature. The various attention mechanisms are explained by means of a framework consisting of a general attention model, uniform notation, and a comprehensive taxonomy of attention mechanisms. Furthermore, the various measures for evaluating attention models are reviewed, and methods to characterize the structure of attention models based on the proposed framework are discussed. Last, future work in the field of attention models is considered.
AbstractList	Attention is an important mechanism that can be employed for a variety of deep learning models across many different domains and tasks. This survey provides an overview of the most important attention mechanisms proposed in the literature. The various attention mechanisms are explained by means of a framework consisting of a general attention model, uniform notation, and a comprehensive taxonomy of attention mechanisms. Furthermore, the various measures for evaluating attention models are reviewed, and methods to characterize the structure of attention models based on the proposed framework are discussed. Last, future work in the field of attention models is considered.
Author	Brauwers, Gianni Frasincar, Flavius
Author_xml	– sequence: 1 givenname: Gianni orcidid: 0000-0001-6550-6588 surname: Brauwers fullname: Brauwers, Gianni email: brauwers@ese.eur.nl organization: Erasmus School of Economics, Erasmus University Rotterdam, Rotterdam, DR, The Netherlands – sequence: 2 givenname: Flavius orcidid: 0000-0002-8031-758X surname: Frasincar fullname: Frasincar, Flavius email: frasincar@ese.eur.nl organization: Erasmus School of Economics, Erasmus University Rotterdam, Rotterdam, DR, The Netherlands
BookMark	eNp9kE1PAjEQhhuDiaD-AONlE8-LnX5ujwQQjRgP4rkpy6yWQBfbxYR_724gHjx4mvfwPjOZZ0B6oQ5IyA3QIQA194vnyXTIKIMhB6aEVGekD1IWOQMDvTZTAbngQl-QQUprSmmhC-gTM8pmGDC6Tfa2j994yOqQjZoGQ-Pb9ILlpws-bVPmQzZB3GVzdDH48HFFziu3SXh9mpfk_WG6GD_m89fZ03g0z0smBMud4eDQFKuVKFGoUgoql1QzRQ1osWIoFUUjoAKlAI2kWrCSSsGrJawqp_gluTvu3cX6a4-pset6H0N70jJdCCkp10Xb0sdWGeuUIla29I3rfmii8xsL1HaebOfJdp7syVNLwh9yF_3WxcO_zO2R8Yj42zftT5Ib_gN2eHJx
CODEN	ITKEEH
CitedBy_id	crossref_primary_10_1016_j_compbiomed_2025_110102 crossref_primary_10_1016_j_compbiomed_2025_110345 crossref_primary_10_1016_j_compositesa_2024_108429 crossref_primary_10_1007_s11760_025_04633_3 crossref_primary_10_3390_rs16071214 crossref_primary_10_1109_ACCESS_2023_3306022 crossref_primary_10_1007_s11004_025_10220_6 crossref_primary_10_1109_ACCESS_2024_3509605 crossref_primary_10_1016_j_oceaneng_2025_120820 crossref_primary_10_1016_j_asoc_2025_112978 crossref_primary_10_3390_app14030978 crossref_primary_10_1109_TITS_2022_3186613 crossref_primary_10_54097_6vbssk40 crossref_primary_10_1364_JOCN_481084 crossref_primary_10_1016_j_phycom_2024_102534 crossref_primary_10_1007_s11071_025_11722_y crossref_primary_10_1016_j_engstruct_2024_118516 crossref_primary_10_1016_j_ejor_2024_08_024 crossref_primary_10_1080_01431161_2023_2285742 crossref_primary_10_3390_e25030440 crossref_primary_10_1109_TII_2024_3523580 crossref_primary_10_1016_j_rineng_2025_104783 crossref_primary_10_1007_s10489_023_05242_8 crossref_primary_10_1186_s40001_025_03066_5 crossref_primary_10_1016_j_eswa_2025_129446 crossref_primary_10_1016_j_susmat_2025_e01535 crossref_primary_10_1177_09544062241305515 crossref_primary_10_1002_ima_70190 crossref_primary_10_1002_sdtp_18953 crossref_primary_10_1007_s10773_022_05209_4 crossref_primary_10_3390_s25185682 crossref_primary_10_1109_TASE_2025_3592721 crossref_primary_10_1007_s11554_024_01575_w crossref_primary_10_1016_j_oceaneng_2025_121929 crossref_primary_10_1109_TGRS_2024_3350573 crossref_primary_10_3390_asi6060115 crossref_primary_10_1016_j_saa_2025_126336 crossref_primary_10_3390_rs17010052 crossref_primary_10_1016_j_procs_2025_03_185 crossref_primary_10_1007_s00521_024_10698_x crossref_primary_10_1109_ACCESS_2023_3296710 crossref_primary_10_1016_j_asoc_2023_110335 crossref_primary_10_1186_s12891_025_08733_6 crossref_primary_10_1016_j_neucom_2024_129255 crossref_primary_10_3390_electronics14142865 crossref_primary_10_46670_JSST_2025_34_4_289 crossref_primary_10_14778_3749646_3749712 crossref_primary_10_1007_s11760_025_04562_1 crossref_primary_10_1109_ACCESS_2024_3513445 crossref_primary_10_3390_s23229155 crossref_primary_10_1016_j_knosys_2025_113538 crossref_primary_10_1016_j_jhydrol_2025_132879 crossref_primary_10_1016_j_patrec_2025_03_024 crossref_primary_10_1177_11779322241272387 crossref_primary_10_1007_s11042_025_20999_5 crossref_primary_10_1109_ACCESS_2025_3546681 crossref_primary_10_3390_electronics13183783 crossref_primary_10_1007_s11771_024_5762_9 crossref_primary_10_1016_j_knosys_2025_113301 crossref_primary_10_1088_1361_6501_ad11c9 crossref_primary_10_1093_gigascience_giae089 crossref_primary_10_1016_j_jfranklin_2025_107511 crossref_primary_10_1002_cpe_70250 crossref_primary_10_1109_TGRS_2024_3374600 crossref_primary_10_3390_rs17060971 crossref_primary_10_1109_JSEN_2024_3400519 crossref_primary_10_1038_s41598_025_16564_7 crossref_primary_10_1016_j_aei_2025_103252 crossref_primary_10_1007_s10489_023_05101_6 crossref_primary_10_1080_1448837X_2025_2557647 crossref_primary_10_4103_ATMR_ATMR_70_25 crossref_primary_10_1016_j_renene_2024_121133 crossref_primary_10_3389_fphys_2025_1512835 crossref_primary_10_1016_j_brainres_2025_149484 crossref_primary_10_1177_09544054251359468 crossref_primary_10_1016_j_measurement_2024_116290 crossref_primary_10_1109_ACCESS_2024_3365349 crossref_primary_10_3390_wevj16090487 crossref_primary_10_1109_TITS_2024_3510678 crossref_primary_10_3390_jmse12060990 crossref_primary_10_1016_j_tws_2024_111797 crossref_primary_10_3390_rs17173026 crossref_primary_10_1016_j_rineng_2024_102397 crossref_primary_10_1038_s41598_025_90607_x crossref_primary_10_1038_s41598_024_52451_3 crossref_primary_10_1016_j_jksuci_2024_102038 crossref_primary_10_1109_JSEN_2024_3524584 crossref_primary_10_1109_ACCESS_2024_3417422 crossref_primary_10_3390_agriculture14020228 crossref_primary_10_1016_j_ins_2025_122609 crossref_primary_10_1109_TIM_2024_3374311 crossref_primary_10_3389_fphys_2024_1337554 crossref_primary_10_1007_s13735_023_00279_4 crossref_primary_10_1016_j_cose_2025_104509 crossref_primary_10_3390_su17072885 crossref_primary_10_3390_electronics14112241 crossref_primary_10_1109_TKDE_2024_3447878 crossref_primary_10_3390_s25133908 crossref_primary_10_4108_eetsis_8701 crossref_primary_10_1007_s12204_024_2694_3 crossref_primary_10_1109_TGRS_2025_3559590 crossref_primary_10_3389_fpsyt_2023_1310323 crossref_primary_10_1109_ACCESS_2024_3389669 crossref_primary_10_1016_j_tws_2024_112878 crossref_primary_10_3390_en18123057 crossref_primary_10_3390_info14110610 crossref_primary_10_1016_j_neunet_2025_107496 crossref_primary_10_1016_j_sna_2024_115521 crossref_primary_10_1007_s10462_023_10577_2 crossref_primary_10_1016_j_ins_2023_119307 crossref_primary_10_1016_j_autcon_2025_106419 crossref_primary_10_1016_j_renene_2025_123774 crossref_primary_10_1016_j_bspc_2025_107886 crossref_primary_10_1016_j_asoc_2025_113692 crossref_primary_10_3389_fpls_2024_1395796 crossref_primary_10_1016_j_asoc_2025_113326 crossref_primary_10_1016_j_medntd_2025_100382 crossref_primary_10_1007_s10586_025_05343_8 crossref_primary_10_3390_sym17060820 crossref_primary_10_3390_electronics13081454 crossref_primary_10_3390_app15126600 crossref_primary_10_3390_electronics14091818 crossref_primary_10_1016_j_eswa_2025_128287 crossref_primary_10_1038_s41598_024_54114_9 crossref_primary_10_1109_TGRS_2024_3456754 crossref_primary_10_1007_s10115_022_01764_8 crossref_primary_10_1109_TKDE_2024_3386895 crossref_primary_10_1016_j_resconrec_2024_107925 crossref_primary_10_1109_LGRS_2022_3225713 crossref_primary_10_3390_pr13082365 crossref_primary_10_3354_meps14842 crossref_primary_10_3390_rs17020307 crossref_primary_10_1109_TKDE_2024_3382726 crossref_primary_10_1016_j_engappai_2025_110971 crossref_primary_10_1088_2632_2153_ad2e15 crossref_primary_10_1016_j_compeleceng_2024_109520 crossref_primary_10_1016_j_microc_2025_113384 crossref_primary_10_1007_s11063_025_11755_9 crossref_primary_10_3390_fishes10040173 crossref_primary_10_1109_JSTARS_2024_3400407 crossref_primary_10_32604_cmc_2023_032364 crossref_primary_10_3390_a17030109 crossref_primary_10_1007_s11760_024_03759_0 crossref_primary_10_1007_s10598_025_09633_z crossref_primary_10_1057_s41599_025_04633_1 crossref_primary_10_1080_01431161_2022_2091964 crossref_primary_10_1021_acsestwater_5c00467 crossref_primary_10_1016_j_solener_2025_113769 crossref_primary_10_1016_j_eswa_2023_122887 crossref_primary_10_3390_ijgi13120433 crossref_primary_10_1016_j_mtcomm_2025_113778 crossref_primary_10_3390_electronics12092032 crossref_primary_10_1371_journal_pone_0319776 crossref_primary_10_1007_s10044_024_01276_x crossref_primary_10_1371_journal_pone_0317355 crossref_primary_10_1016_j_apor_2024_104072 crossref_primary_10_3390_app14052144 crossref_primary_10_1002_ird3_115 crossref_primary_10_1109_TIM_2025_3561378 crossref_primary_10_1038_s41598_025_85139_3 crossref_primary_10_1088_1361_6560_ad8290 crossref_primary_10_3390_computers12080151 crossref_primary_10_1016_j_aei_2024_102925 crossref_primary_10_1007_s12194_024_00875_x crossref_primary_10_1016_j_nlp_2025_100168 crossref_primary_10_3390_agriculture14020273 crossref_primary_10_3390_app132212421 crossref_primary_10_1109_TGRS_2023_3339291 crossref_primary_10_1016_j_eswa_2025_126691 crossref_primary_10_1371_journal_pone_0312474 crossref_primary_10_1109_TITS_2025_3545942 crossref_primary_10_3390_agronomy13051240 crossref_primary_10_1016_j_eswa_2024_125779 crossref_primary_10_1016_j_knosys_2025_113825 crossref_primary_10_3390_agronomy15020489 crossref_primary_10_1109_TAES_2025_3564932 crossref_primary_10_1007_s11042_024_19479_z crossref_primary_10_3390_en17071658 crossref_primary_10_3390_ani13223535 crossref_primary_10_1007_s11053_022_10142_8 crossref_primary_10_3390_machines12070492 crossref_primary_10_1016_j_jnca_2024_103903 crossref_primary_10_1080_07421222_2024_2376384 crossref_primary_10_3390_rs16111860 crossref_primary_10_1080_10400419_2024_2361217 crossref_primary_10_1007_s12083_025_02093_7 crossref_primary_10_1016_j_knosys_2024_111457 crossref_primary_10_7717_peerj_cs_2981 crossref_primary_10_1016_j_energy_2025_136265 crossref_primary_10_3390_rs17122058 crossref_primary_10_1016_j_ymssp_2025_113212 crossref_primary_10_1088_1361_6501_ada6ee crossref_primary_10_1080_10589759_2024_2386349 crossref_primary_10_20965_jaciii_2025_p0868 crossref_primary_10_1088_1402_4896_ad7bfd crossref_primary_10_3390_en18112902 crossref_primary_10_1016_j_csl_2024_101751 crossref_primary_10_1016_j_eswa_2024_125525 crossref_primary_10_1117_1_JRS_19_026503 crossref_primary_10_3233_JIFS_240531 crossref_primary_10_3389_frai_2025_1579998 crossref_primary_10_1117_1_JRS_19_026506 crossref_primary_10_1016_j_ecoinf_2024_102922 crossref_primary_10_1109_ACCESS_2025_3549796 crossref_primary_10_1109_ACCESS_2024_3458908 crossref_primary_10_1016_j_vehcom_2025_100895 crossref_primary_10_1002_adts_202400479 crossref_primary_10_1007_s40430_025_05457_5 crossref_primary_10_1177_14727978251380830 crossref_primary_10_3390_app15010146 crossref_primary_10_1515_revneuro_2024_0088 crossref_primary_10_1016_j_jenvman_2025_125094 crossref_primary_10_1007_s11144_024_02695_x crossref_primary_10_1007_s10489_023_04848_2 crossref_primary_10_1016_j_asoc_2024_112262 crossref_primary_10_1016_j_asoc_2025_113750 crossref_primary_10_1016_j_jpowsour_2024_235674 crossref_primary_10_1016_j_jhydrol_2025_133237 crossref_primary_10_1109_TGRS_2025_3594760 crossref_primary_10_3390_biomimetics10010034 crossref_primary_10_1109_ACCESS_2024_3520608 crossref_primary_10_3390_mi14040893 crossref_primary_10_1038_s41598_024_57321_6 crossref_primary_10_1109_JIOT_2025_3576567 crossref_primary_10_3390_s23115204 crossref_primary_10_1007_s10032_025_00545_7 crossref_primary_10_3390_s23104681 crossref_primary_10_3390_f16071192 crossref_primary_10_1109_OJCS_2025_3581950 crossref_primary_10_32604_cmc_2024_054360 crossref_primary_10_1016_j_datak_2024_102332 crossref_primary_10_1016_j_frl_2024_105846 crossref_primary_10_1155_hbe2_5979728 crossref_primary_10_1016_j_eswa_2025_128678 crossref_primary_10_3390_e27040368 crossref_primary_10_1007_s13349_024_00848_z crossref_primary_10_1186_s13677_023_00483_x crossref_primary_10_1109_JSTARS_2023_3339294 crossref_primary_10_1016_j_imavis_2024_104926 crossref_primary_10_1016_j_engappai_2023_107270 crossref_primary_10_1109_TVT_2023_3304530 crossref_primary_10_1016_j_ecoinf_2024_102904 crossref_primary_10_1021_jacs_4c10893 crossref_primary_10_3390_electronics13030539 crossref_primary_10_1016_j_apenergy_2024_123947 crossref_primary_10_1162_imag_a_00214 crossref_primary_10_1140_epjp_s13360_024_05287_9 crossref_primary_10_3390_app14156670 crossref_primary_10_1007_s42452_025_06766_z crossref_primary_10_1016_j_geoen_2025_213762 crossref_primary_10_1109_TIM_2025_3568977 crossref_primary_10_1007_s10489_023_04571_y crossref_primary_10_1007_s11030_024_11100_7 crossref_primary_10_3390_a17110488 crossref_primary_10_1016_j_apenergy_2025_126368 crossref_primary_10_1109_TGRS_2023_3293832 crossref_primary_10_3389_fnins_2022_920820 crossref_primary_10_3390_ijgi13070222 crossref_primary_10_1016_j_jclepro_2025_145873 crossref_primary_10_1093_comjnl_bxae132 crossref_primary_10_1287_isre_2022_0529 crossref_primary_10_3390_wevj16090541 crossref_primary_10_1109_JSEN_2024_3372298 crossref_primary_10_1007_s00521_025_11557_z crossref_primary_10_3390_app15010212 crossref_primary_10_1109_ACCESS_2023_3339560 crossref_primary_10_2478_amns_2025_0210 crossref_primary_10_3390_e27040348 crossref_primary_10_1016_j_eswa_2025_126837 crossref_primary_10_1109_ACCESS_2025_3588407 crossref_primary_10_1016_j_ecoinf_2024_102683 crossref_primary_10_3390_rs15215227 crossref_primary_10_1109_JBHI_2024_3355951 crossref_primary_10_3390_s24092925 crossref_primary_10_1007_s10489_022_04048_4 crossref_primary_10_3390_diagnostics14040390 crossref_primary_10_1109_JSEN_2025_3582423 crossref_primary_10_1016_j_ocecoaman_2025_107841 crossref_primary_10_1109_TAES_2024_3449268 crossref_primary_10_1016_j_compag_2023_108352 crossref_primary_10_1016_j_cviu_2025_104304 crossref_primary_10_1016_j_engappai_2025_110218 crossref_primary_10_3390_info16070602 crossref_primary_10_3390_math13152437 crossref_primary_10_1109_LSP_2024_3398358 crossref_primary_10_1088_1674_1056_adce96
Cites_doi	10.1109/CVPR.2018.00911 10.1609/aaai.v32i1.12055 10.1016/j.jvcir.2020.102775 10.1145/3357384.3357895 10.1007/978-3-319-50835-1_22 10.1007/978-3-540-30120-2_59 10.18653/v1/n16-1174 10.1007/bf00992696 10.24963/ijcai.2019/513 10.1007/s10115-020-01513-9 10.18653/v1/D19-1002 10.1109/ICDAR.2019.00061 10.1007/978-3-030-21348-0_24 10.18653/v1/W19-4320 10.18653/v1/D15-1166 10.1016/j.cviu.2017.10.001 10.1007/978-3-030-00937-3_43 10.1109/TNNLS.2018.2869225 10.18653/v1/d18-1380 10.1007/978-3-030-21074-8_4 10.24963/ijcai.2018/546 10.18653/v1/D18-1176 10.1111/coin.12225 10.1109/ICASSP.2016.7472618 10.3115/v1/W14-4012 10.1109/CVPR42600.2020.01009 10.1109/CVPR.2018.00636 10.1109/ICASSP.2019.8683483 10.1109/TMM.2017.2729019 10.18653/v1/D19-1671 10.1109/CVPR.2016.10 10.1609/aaai.v34i05.6250 10.1109/CVPR42600.2020.01059 10.1609/aaai.v32i1.11965 10.1609/aaai.v31i1.11197 10.1109/ICDM.2019.00120 10.18653/v1/P19-1260 10.1109/TKDE.2019.2913394 10.18653/v1/D19-1016 10.21437/Interspeech.2019-2616 10.1007/978-3-030-32236-6_16 10.18653/v1/P18-1162 10.1145/3178876.3186015 10.24963/ijcai.2017/568 10.1007/978-3-030-29513-4_31 10.1145/3465055 10.18653/v1/D17-1151 10.1109/CVPR.2017.648 10.18653/v1/d16-1058 10.1109/TPAMI.2018.2889052 10.1145/3336191.3371845 10.1145/3308558.3313513 10.1145/3308558.3313750 10.1145/3209978.3210009 10.3115/1073083.1073135 10.1609/aaai.v34i05.6211 10.1109/JBHI.2020.2986926 10.1109/TIP.2019.2928634 10.1146/annurev.neuro.26.041002.131047 10.1609/aaai.v32i1.11941 10.3115/1626355.1626362 10.18653/v1/p17-1164 10.18653/v1/2020.acl-main.48 10.18653/v1/W18-5429 10.1145/3530811 10.18653/v1/P19-1345 10.1609/aaai.v32i1.11254 10.1109/ICASSP.2019.8682539 10.1145/3219819.3220086 10.18653/v1/D18-1375 10.1609/aaai.v32i1.11635 10.21437/Interspeech.2017-486 10.1609/aaai.v33i01.33018658 10.1145/3109859.3109890 10.1016/j.image.2003.09.001 10.1609/aaai.v33i01.33019324 10.1609/aaai.v32i1.12048 10.18653/v1/p17-1036 10.1109/ICASSP.2017.7953075 10.1145/3363574 10.24963/ijcai.2018/604 10.18653/v1/w18-2601 10.18653/v1/P18-1240 10.18653/v1/P19-2030 10.18653/v1/2020.acl-main.387 10.1007/978-3-030-01231-1_1 10.18653/v1/P19-1285
ContentType	Journal Article
Copyright	Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023
Copyright_xml	– notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023
DBID	97E RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D
DOI	10.1109/TKDE.2021.3126456
DatabaseName	IEEE Xplore (IEEE) IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional
DatabaseTitle	CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional
DatabaseTitleList	Technology Research Database
Database_xml	– sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher
DeliveryMethod	fulltext_linktorsrc
Discipline	Engineering Computer Science
EISSN	1558-2191
EndPage	3298
ExternalDocumentID	10_1109_TKDE_2021_3126456 9609539
Genre	orig-research
GroupedDBID	-~X .DC 0R~ 29I 4.4 5GY 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFO ACIWK AENEX AGQYO AHBIQ AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ASUFR ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 EBS EJD F5P HZ~ IEDLZ IFIPE IPLJI JAVBF LAI M43 MS~ O9- OCL P2P PQQKQ RIA RIE RNS RXW TAE TN5 UHB AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D
ID	FETCH-LOGICAL-c2442-a931ae98dd4ce46c5405b072609174d2e560e941f1661e950742c0543fb1dfa63
IEDL.DBID	RIE
ISSN	1041-4347
IngestDate	Mon Jun 30 06:17:44 EDT 2025 Tue Nov 18 22:27:35 EST 2025 Sat Nov 29 02:36:04 EST 2025 Wed Aug 27 02:18:06 EDT 2025
IsPeerReviewed	true
IsScholarly	true
Issue	4
Language	English
License	https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c2442-a931ae98dd4ce46c5405b072609174d2e560e941f1661e950742c0543fb1dfa63
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
ORCID	0000-0001-6550-6588 0000-0002-8031-758X
PQID	2784550378
PQPubID	85438
PageCount	20
ParticipantIDs	proquest_journals_2784550378 crossref_primary_10_1109_TKDE_2021_3126456 ieee_primary_9609539 crossref_citationtrail_10_1109_TKDE_2021_3126456
PublicationCentury	2000
PublicationDate	2023-04-01
PublicationDateYYYYMMDD	2023-04-01
PublicationDate_xml	– month: 04 year: 2023 text: 2023-04-01 day: 01
PublicationDecade	2020
PublicationPlace	New York
PublicationPlace_xml	– name: New York
PublicationTitle	IEEE transactions on knowledge and data engineering
PublicationTitleAbbrev	TKDE
PublicationYear	2023
Publisher	IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml	– name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References	Daniluk (ref23) ref57 Mnih (ref2) ref56 ref59 Yu (ref106) ref58 ref53 ref52 Xu (ref8) Tay (ref103) 2021; 139 ref51 ref50 Yu (ref25) 2018 Wang (ref46) 2016 ref45 ref48 ref47 Alami Mejjati (ref125) ref42 Ndajah (ref113) ref41 ref44 ref43 Veličković (ref31) ref49 Zhang (ref74) ref7 Veličković (ref83) ref9 ref4 ref3 Vaswani (ref13) ref6 ref5 Zhang (ref80) 2018 ref40 ref34 ref37 Lin (ref71) ref30 ref33 ref39 ref38 Chen (ref115) 2020 Wang (ref101) 2020 Ba (ref72) 2016 ref24 Graves (ref55) 2014 ref20 Chen (ref95) 2021 Chorowski (ref10) ref21 ref28 ref27 ref29 Parmar (ref15) Larochelle (ref1) Sordoni (ref54) 2016 ref12 Jain (ref120) 2019 ref14 ref97 ref126 ref96 ref11 ref99 ref124 ref98 Zheng (ref36) 2018 ref17 ref16 ref19 Lu (ref32) ref93 ref92 ref94 ref91 ref89 ref86 ref85 ref88 ref87 Banerjee (ref108) Wang (ref18) 2016 Galassi (ref22) 2019 Sennrich (ref109) Oktay (ref67) ref82 ref81 ref84 ref79 ref78 ref107 ref104 ref105 ref77 ref76 Sharma (ref26) 2016 Kitaev (ref100) ref111 ref70 ref112 ref73 ref110 ref68 ref119 ref117 ref69 ref118 ref64 ref63 ref116 ref66 ref65 ref114 Seo (ref35) Sabour (ref90) 2017 ref60 Goodfellow (ref75) ref122 Wu (ref102) ref62 ref61 ref121 Thekumparampil (ref123) 2018
References_xml	– ident: ref16 doi: 10.1109/CVPR.2018.00911 – ident: ref47 doi: 10.1609/aaai.v32i1.12055 – ident: ref63 doi: 10.1016/j.jvcir.2020.102775 – ident: ref17 doi: 10.1145/3357384.3357895 – ident: ref114 doi: 10.1007/978-3-319-50835-1_22 – volume-title: Proc. 6th Int. Conf. Learn. Representations ident: ref31 article-title: Graph attention networks – ident: ref111 doi: 10.1007/978-3-540-30120-2_59 – ident: ref5 doi: 10.18653/v1/n16-1174 – year: 2014 ident: ref55 article-title: Neural Turing machines – ident: ref57 doi: 10.1007/bf00992696 – ident: ref70 doi: 10.24963/ijcai.2019/513 – ident: ref93 doi: 10.1007/s10115-020-01513-9 – ident: ref121 doi: 10.18653/v1/D19-1002 – ident: ref65 doi: 10.1109/ICDAR.2019.00061 – ident: ref43 doi: 10.1007/978-3-030-21348-0_24 – ident: ref52 doi: 10.18653/v1/W19-4320 – start-page: 65 volume-title: Proc. Workshop Intrinsic Extrinsic Eval. Measures Mach. Transl. Summarization ident: ref108 article-title: METEOR: An automatic metric for MT evaluation with improved correlation with human judgments – start-page: 577 volume-title: Proc. 28th Annu. Conf. Neural Inf. Process. Syst. ident: ref10 article-title: Attention-based models for speech recognition – ident: ref4 doi: 10.18653/v1/D15-1166 – ident: ref118 doi: 10.1016/j.cviu.2017.10.001 – ident: ref124 doi: 10.1007/978-3-030-00937-3_43 – start-page: 4055 volume-title: Proc. 35th Int. Conf. Mach. Learn. ident: ref15 article-title: Image Transformer – ident: ref30 doi: 10.1109/TNNLS.2018.2869225 – ident: ref33 doi: 10.18653/v1/d18-1380 – ident: ref77 doi: 10.1007/978-3-030-21074-8_4 – year: 2020 ident: ref101 article-title: Linformer: Self-attention with linear complexity – ident: ref28 doi: 10.24963/ijcai.2018/546 – year: 2018 ident: ref36 article-title: Left-center-right separated neural network for aspect-based sentiment analysis with rotatory attention – ident: ref50 doi: 10.18653/v1/D18-1176 – ident: ref53 doi: 10.1111/coin.12225 – ident: ref11 doi: 10.1109/ICASSP.2016.7472618 – ident: ref14 doi: 10.3115/v1/W14-4012 – ident: ref73 doi: 10.1109/CVPR42600.2020.01009 – start-page: 539 volume-title: Proc. 13th Conf. Eur. Chapter Assoc. Comput. Linguistics ident: ref109 article-title: Perplexity minimization for translation model domain adaptation in statistical machine translation – ident: ref7 doi: 10.1109/CVPR.2018.00636 – ident: ref87 doi: 10.1109/ICASSP.2019.8683483 – volume: 139 start-page: 10183 year: 2021 ident: ref103 article-title: Synthesizer: Rethinking self-attention for transformer models publication-title: Proc. 38th Int. Conf. Mach. Learn. – volume-title: Proc. 5th Int. Conf. Learn. Representations ident: ref23 article-title: Frustratingly short attention spans in neural language modeling – ident: ref27 doi: 10.1109/TMM.2017.2729019 – ident: ref92 doi: 10.18653/v1/D19-1671 – ident: ref88 doi: 10.1109/CVPR.2016.10 – ident: ref69 doi: 10.1609/aaai.v34i05.6250 – start-page: 2672 volume-title: Proc. 27th Annu. Conf. Neural Inf. Process. Syst. ident: ref75 article-title: Generative adversarial nets – ident: ref94 doi: 10.1109/CVPR42600.2020.01059 – volume-title: Proc. 4th Int. Conf. Learn. Representations ident: ref35 article-title: Bidirectional attention flow for machine comprehension – ident: ref48 doi: 10.1609/aaai.v32i1.11965 – ident: ref117 doi: 10.1609/aaai.v31i1.11197 – volume-title: Proc. 5th Int. Conf. Learn. Representations ident: ref71 article-title: A structured self-attentive sentence embedding – ident: ref38 doi: 10.1109/ICDM.2019.00120 – year: 2019 ident: ref22 article-title: Attention, please! a critical review of neural attention models in natural language processing – year: 2018 ident: ref123 article-title: Attention-based graph neural network for semi-supervised learning – ident: ref41 doi: 10.18653/v1/P19-1260 – volume-title: Proc. 1st Med. Imag. Deep Learn. Conf. ident: ref67 article-title: Attention U-Net: Learning where to look for the pancreas – start-page: 2204 volume-title: Proc. 27th Annu. Conf. Neural Inf. Process. Syst. ident: ref2 article-title: Recurrent models of visual attention – ident: ref45 doi: 10.1109/TKDE.2019.2913394 – ident: ref96 doi: 10.18653/v1/D19-1016 – ident: ref91 doi: 10.21437/Interspeech.2019-2616 – ident: ref68 doi: 10.1007/978-3-030-32236-6_16 – ident: ref66 doi: 10.18653/v1/P18-1162 – start-page: 3693 volume-title: Proc. 32nd Annu. Conf. Neural Inf. Process. Syst. ident: ref125 article-title: Unsupervised attention-guided image-to-image translation – ident: ref89 doi: 10.1145/3178876.3186015 – year: 2018 ident: ref80 article-title: Next item recommendation with self-attention – start-page: 1243 volume-title: Proc. 24th Annu. Conf. Neural Inf. Process. Syst. ident: ref1 article-title: Learning to combine foveal glimpses with a third-order Boltzmann machine – ident: ref34 doi: 10.24963/ijcai.2017/568 – ident: ref21 doi: 10.1007/978-3-030-29513-4_31 – ident: ref20 doi: 10.1145/3465055 – year: 2020 ident: ref115 article-title: Residual attention U-net for automated multi-class segmentation of COVID-19 chest CT images – ident: ref56 doi: 10.18653/v1/D17-1151 – ident: ref119 doi: 10.1109/CVPR.2017.648 – ident: ref6 doi: 10.18653/v1/d16-1058 – start-page: 7354 volume-title: Proc. 36th Int. Conf. Mach. Learn. ident: ref74 article-title: Self-attention generative adversarial networks – ident: ref79 doi: 10.1109/TPAMI.2018.2889052 – ident: ref82 doi: 10.1145/3336191.3371845 – ident: ref40 doi: 10.1145/3308558.3313513 – ident: ref98 doi: 10.1145/3308558.3313750 – ident: ref85 doi: 10.1145/3209978.3210009 – ident: ref107 doi: 10.3115/1073083.1073135 – ident: ref60 doi: 10.1609/aaai.v34i05.6211 – start-page: 5998 volume-title: Proc. 31st Annu. Conf. Neural Inf. Process. Syst. ident: ref13 article-title: Attention is all you need – ident: ref76 doi: 10.1109/JBHI.2020.2986926 – ident: ref49 doi: 10.1109/TIP.2019.2928634 – ident: ref3 doi: 10.1146/annurev.neuro.26.041002.131047 – ident: ref64 doi: 10.1609/aaai.v32i1.11941 – start-page: 53 volume-title: Proc. 3rd WSEAS Int. Conf. Vis. Imaging Simul. ident: ref113 article-title: SSIM image quality metric for denoised images – volume-title: Proc. 6th Int. Conf. Learn. Representations ident: ref106 article-title: QANet: Combining local convolution with global self-attention for reading comprehension – ident: ref110 doi: 10.3115/1626355.1626362 – ident: ref116 doi: 10.18653/v1/p17-1164 – ident: ref42 doi: 10.18653/v1/2020.acl-main.48 – start-page: 289 volume-title: Proc. 30th Annu. Conf. Neural Inf. Process. Syst. ident: ref32 article-title: Hierarchical question-image co-attention for visual question answering – ident: ref81 doi: 10.18653/v1/W18-5429 – ident: ref104 doi: 10.1145/3530811 – ident: ref62 doi: 10.18653/v1/P19-1345 – year: 2016 ident: ref46 article-title: Hierarchical attention network for action recognition in videos – ident: ref44 doi: 10.1609/aaai.v32i1.11254 – year: 2021 ident: ref95 article-title: TransUnet: Transformers make strong encoders for medical image segmentation – ident: ref78 doi: 10.1109/ICASSP.2019.8682539 – ident: ref39 doi: 10.1145/3219819.3220086 – ident: ref51 doi: 10.18653/v1/D18-1375 – ident: ref29 doi: 10.1609/aaai.v32i1.11635 – start-page: 3543 year: 2019 ident: ref120 article-title: Attention is not explanation publication-title: Proc. Conf. North Amer. Chapter Assoc. Comput. Linguistics: Hum. Language Technol. – year: 2016 ident: ref18 article-title: Survey on the attention based RNN model and its applications in computer vision – ident: ref24 doi: 10.21437/Interspeech.2017-486 – ident: ref105 doi: 10.1609/aaai.v33i01.33018658 – ident: ref61 doi: 10.1145/3109859.3109890 – ident: ref112 doi: 10.1016/j.image.2003.09.001 – ident: ref97 doi: 10.1609/aaai.v33i01.33019324 – ident: ref9 doi: 10.1609/aaai.v32i1.12048 – ident: ref126 doi: 10.18653/v1/p17-1036 – ident: ref12 doi: 10.1109/ICASSP.2017.7953075 – ident: ref19 doi: 10.1145/3363574 – ident: ref58 doi: 10.24963/ijcai.2018/604 – start-page: 2048 volume-title: Proc. 32nd Int. Conf. Mach. Learn. ident: ref8 article-title: Show, attend and tell: Neural image caption generation with visual attention – volume-title: Proc. 8th Int. Conf. Learn. Representations ident: ref102 article-title: Lite transformer with long-short range attention – start-page: 3859 year: 2017 ident: ref90 article-title: Dynamic routing between capsules publication-title: Proc. 31st Annu. Conf. Neural Inf. Process. Syst. – year: 2016 ident: ref72 article-title: Layer normalization – ident: ref86 doi: 10.18653/v1/w18-2601 – ident: ref37 doi: 10.18653/v1/P18-1240 – year: 2016 ident: ref54 article-title: Iterative alternating neural attention for machine reading – volume-title: Proc. 5th Int. Conf. Learn. Representations ident: ref83 article-title: Graph attention networks – ident: ref84 doi: 10.18653/v1/P19-2030 – volume-title: Proc. 8th Int. Conf. Learn. Representations ident: ref100 article-title: Reformer: The efficient Transformer – ident: ref122 doi: 10.18653/v1/2020.acl-main.387 – start-page: 188 year: 2018 ident: ref25 article-title: Multi-level attention model for weakly supervised audio classification publication-title: Proc. Detection Classification Acoustic Scenes Events Workshop – ident: ref59 doi: 10.1007/978-3-030-01231-1_1 – year: 2016 ident: ref26 article-title: Action recognition using visual attention publication-title: Proc. 4th Int. Conf. Learn. Representations Workshop – ident: ref99 doi: 10.18653/v1/P19-1285
SSID	ssj0008781
Score	2.7414463
Snippet	Attention is an important mechanism that can be employed for a variety of deep learning models across many different domains and tasks. This survey provides an...
SourceID	proquest crossref ieee
SourceType	Aggregation Database Enrichment Source Index Database Publisher
StartPage	3279
SubjectTerms	Attention models Biological system modeling Computational modeling Data models Deep learning Feature extraction introductory and survey neural nets supervised learning Task analysis Taxonomy
Title	A General Survey on Attention Mechanisms in Deep Learning
URI	https://ieeexplore.ieee.org/document/9609539 https://www.proquest.com/docview/2784550378
Volume	35
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 1558-2191 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0008781 issn: 1041-4347 databaseCode: RIE dateStart: 19890101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1BS8MwFH5sw4MenG6K0yk5eBLr1jZrmuNwG4I6BCfsVtr0VQbaja0b-O99abMyUARvPSRQ3tfkfV-T9z6Aa49IiBKOsgQSDJyH3Ap9rVodobxQROiJvFD4SYzH_nQqXypwW9bCIGJ--Qzv9GN-lh_P1Vr_KusU3dFkFapCeEWtVrnr-iI3JCV1QZrI5cKcYNpd2Zk8DoakBB2bBCrlf-1VvZODclOVHztxnl5G9f-92BEcGhrJ-gXux1DBtAH1rUUDMyu2AQc7_QabIPvMtJlmr-vlBr_YPGX9LCuuPLJn1GXAs9Xnis1SNkBcMNN-9f0E3kbDyf2DZbwTLEUJ27FC6dohSj-OuULuKU3Moq4g9UL6jMcOEtNBye3EpgSNsqclsiL65iaRHSeh555CLZ2neAYs6kVhTCwq9DnJKZVErh9JghaVS3TNVS3obqMZKNNYXPtbfAS5wOjKQAMQaAACA0ALbsopi6Krxl-Dmzri5UAT7Ba0t5AFZt2tAn2MSprLFf7577MuYF8bxhd3b9pQy5ZrvIQ9tclmq-VV_kl9A4YwxYo
linkProvider	IEEE
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NS8NAEB1qFdSD1apYrboHT2Jsk2yz2WOxLZV-IFiht5BsplLQtPQL_PfOJttSUARvOexCmJfdeS-7Mw_gziMSooSjLIEEA-cht0Jfq1ZHKC8UEXoiLRTuin7fHw7lSw4eNrUwiJhePsNH_Zie5ccTtdS_yipZdzS5A7vaOctUa232XV-klqSkL0gVuVyYM0y7KiuDTqNJWtCxSaISA9Bu1VtZKLVV-bEXpwmmVfjfqx3DkSGSrJ4hfwI5TIpQWJs0MLNmi3C41XHwFGSdmUbT7HU5W-EXmySsvlhklx5ZD3Uh8Hj-OWfjhDUQp8w0YH0_g7dWc_DUtox7gqUoZTtWKF07ROnHMVfIPaWpWVQVpF9IofHYQeI6KLk9silFo6xpkayIwLmjyI5HoeeeQz6ZJHgBLKpFYUw8KvQ5CSo1ilw_kgQuKpcIm6tKUF1HM1Cmtbh2uPgIUolRlYEGINAABAaAEtxvpkyzvhp_DT7VEd8MNMEuQXkNWWBW3jzQB6mkulzhX_4-6xb224NeN-g-9ztXcKDt47ObOGXIL2ZLvIY9tVqM57Ob9PP6Bv_SyNM
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+General+Survey+on+Attention+Mechanisms+in+Deep+Learning&rft.jtitle=IEEE+transactions+on+knowledge+and+data+engineering&rft.au=Brauwers%2C+Gianni&rft.au=Frasincar%2C+Flavius&rft.date=2023-04-01&rft.pub=IEEE&rft.issn=1041-4347&rft.volume=35&rft.issue=4&rft.spage=3279&rft.epage=3298&rft_id=info:doi/10.1109%2FTKDE.2021.3126456&rft.externalDocID=9609539
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1041-4347&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1041-4347&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1041-4347&client=summon