TAP with ease: a generic recommendation system for trigger-action programming based on multi-modal representation learning
The escalating popularity of smart devices has given rise to an increasing trend wherein users leverage customized trigger-action programming (TAP) rules within the Internet of Things (IoT) to automate various aspects of their lives. This article addresses the challenge of effectively combining func...
Uloženo v:
| Vydáno v: | Applied soft computing Ročník 166; s. 112163 |
|---|---|
| Hlavní autoři: | , , |
| Médium: | Journal Article |
| Jazyk: | angličtina |
| Vydáno: |
Elsevier B.V
01.11.2024
|
| Témata: | |
| ISSN: | 1568-4946 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | The escalating popularity of smart devices has given rise to an increasing trend wherein users leverage customized trigger-action programming (TAP) rules within the Internet of Things (IoT) to automate various aspects of their lives. This article addresses the challenge of effectively combining functions provided by many smart devices and online services by introducing a novel multi-modal representation learning model called TAP-TAG. This model integrates both textual and graph structures inherent in TAP rules, offering a holistic method to rule recommendation. TAP-TAG comprises two branches: the Knowledge Graph Embedding model, which projects triplets extracted from the TAP dataset into embeddings, and convolution neural networks that extract semantic features from the textual content of TAP rules. Extensive experiments are conducted on real-world TAP datasets to evaluate our model’s ability to recommend relevant rules based on user preferences. The experimental results show that TAP-TAG can outperform the state-of-the-art method by 5% in Precision@5, indicating that TAP-TAG is highly effective in providing accurate and diverse recommendations for TAP rules.
•Present a multimodal data representation framework for recommending TAP rules.•TAP rules are divided into semantic and relation levels based on their modality.•Improve the representation quality by training text and graph modalities jointly.•Extensive experiments verify the effectiveness of the proposed framework. |
|---|---|
| AbstractList | The escalating popularity of smart devices has given rise to an increasing trend wherein users leverage customized trigger-action programming (TAP) rules within the Internet of Things (IoT) to automate various aspects of their lives. This article addresses the challenge of effectively combining functions provided by many smart devices and online services by introducing a novel multi-modal representation learning model called TAP-TAG. This model integrates both textual and graph structures inherent in TAP rules, offering a holistic method to rule recommendation. TAP-TAG comprises two branches: the Knowledge Graph Embedding model, which projects triplets extracted from the TAP dataset into embeddings, and convolution neural networks that extract semantic features from the textual content of TAP rules. Extensive experiments are conducted on real-world TAP datasets to evaluate our model’s ability to recommend relevant rules based on user preferences. The experimental results show that TAP-TAG can outperform the state-of-the-art method by 5% in Precision@5, indicating that TAP-TAG is highly effective in providing accurate and diverse recommendations for TAP rules.
•Present a multimodal data representation framework for recommending TAP rules.•TAP rules are divided into semantic and relation levels based on their modality.•Improve the representation quality by training text and graph modalities jointly.•Extensive experiments verify the effectiveness of the proposed framework. |
| ArticleNumber | 112163 |
| Author | Wu, Gang Wang, Ming Wang, Feng |
| Author_xml | – sequence: 1 givenname: Gang surname: Wu fullname: Wu, Gang – sequence: 2 givenname: Ming surname: Wang fullname: Wang, Ming – sequence: 3 givenname: Feng surname: Wang fullname: Wang, Feng email: wangfeng12@mails.jlu.edu.cn |
| BookMark | eNp9kMtOwzAQRb0oEm3hB1j5B1LsvIPYVBUvCQkWZW0540lwFduVbUDl60kJKxZdjTQz52rmLMjMOouEXHG24oyX17uVDA5WKUvzFecpL7MZmfOirJO8yctzsghhx8bFJq3n5Hu7fqVfOr5TlAFvqKQ9WvQaqEdwxqBVMmpnaTiEiIZ2ztPodd-jTyT8Tvbe9V4ao21P2zFE0bFpPoaoE-OUHMakvceANk5JA0pvx-ULctbJIeDlX12St_u77eYxeX55eNqsnxPIGItJxVuZNTmDNisUq4qq5E3WVp0EYFnX5HVVFy3kaQPQoeK8akBVWLSdysu2KDBbknTKBe9C8NiJvddG-oPgTByNiZ04GhNHY2IyNkL1Pwj0dH_0Ug-n0dsJxfGpT41eBNBoAZUenUahnD6F_wDxb45M |
| CitedBy_id | crossref_primary_10_1016_j_neucom_2025_131529 crossref_primary_10_1109_JIOT_2025_3532977 crossref_primary_10_1145_3734863 crossref_primary_10_1016_j_eswa_2024_126198 |
| Cites_doi | 10.1109/CVPR.2018.00601 10.1016/j.ipm.2021.102709 10.1145/3576842.3582328 10.1145/3344211 10.1145/3524610.3527922 10.3115/v1/D14-1167 10.1016/j.neucom.2021.10.050 10.1145/3131365.3131369 10.1016/j.patcog.2019.01.006 10.1145/3219819.3220023 10.1016/j.eswa.2023.121065 10.1109/CVPR52729.2023.00859 10.1109/TII.2021.3092774 10.1109/JIOT.2019.2962630 10.1016/j.ipm.2022.102869 10.1109/TII.2021.3128240 10.1145/3290605.3300618 10.1145/3240323.3240377 10.3390/electronics9050750 10.1109/TIP.2023.3331309 10.1016/j.neucom.2021.03.122 10.1145/3447264 10.1109/MC.2017.4041355 10.1016/j.knosys.2022.108859 10.1609/aaai.v37i9.26283 10.1016/j.ins.2021.09.006 10.1109/ICCV.2013.261 10.1109/TIP.2020.3043125 10.1109/CVPR52729.2023.00572 10.1109/TSC.2021.3098756 10.1016/j.neunet.2022.05.026 10.1109/CVPR.2018.00911 10.1145/2962719 10.3115/v1/P15-1085 10.1109/CVPR.2019.00850 10.1016/j.dib.2020.106632 10.1109/JIOT.2019.2940709 10.1145/3290605.3300782 10.1109/JIOT.2018.2866328 10.1109/ACCESS.2019.2903310 |
| ContentType | Journal Article |
| Copyright | 2024 Elsevier B.V. |
| Copyright_xml | – notice: 2024 Elsevier B.V. |
| DBID | AAYXX CITATION |
| DOI | 10.1016/j.asoc.2024.112163 |
| DatabaseName | CrossRef |
| DatabaseTitle | CrossRef |
| DatabaseTitleList | |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| ExternalDocumentID | 10_1016_j_asoc_2024_112163 S1568494624009372 |
| GroupedDBID | --K --M .DC .~1 0R~ 1B1 1~. 1~5 23M 4.4 457 4G. 53G 5GY 5VS 6J9 7-5 71M 8P~ AABNK AACTN AAEDT AAEDW AAIKJ AAKOC AALRI AAOAW AAQFI AAQXK AAXKI AAXUO AAYFN ABBOA ABFNM ABFRF ABJNI ABMAC ABXDB ACDAQ ACGFO ACGFS ACNNM ACRLP ACZNC ADBBV ADEZE ADJOM ADMUD ADTZH AEBSH AECPX AEFWE AEKER AENEX AFJKZ AFKWA AFTJW AGHFR AGUBO AGYEJ AHJVU AHZHX AIALX AIEXJ AIKHN AITUG AJOXV AKRWK ALMA_UNASSIGNED_HOLDINGS AMFUW AMRAJ AOUOD ASPBG AVWKF AXJTR AZFZN BJAXD BKOJK BLXMC CS3 EBS EFJIC EJD EO8 EO9 EP2 EP3 F5P FDB FEDTE FGOYB FIRID FNPLU FYGXN G-Q GBLVA GBOLZ HVGLF HZ~ IHE J1W JJJVA KOM M41 MO0 N9A O-L O9- OAUVE OZT P-8 P-9 P2P PC. Q38 R2- RIG ROL RPZ SDF SDG SES SEW SPC SPCBC SST SSV SSZ T5K UHS UNMZH ~G- 9DU AATTM AAYWO AAYXX ABWVN ACLOT ACRPL ACVFH ADCNI ADNMO AEIPS AEUPX AFPUW AGQPQ AIGII AIIUN AKBMS AKYEP ANKPU APXCP CITATION EFKBS EFLBG ~HD |
| ID | FETCH-LOGICAL-c300t-71ba3940cb35d07576193b7facc03f948785bc429ccfed1179cd7e5bfd46b55e3 |
| ISICitedReferencesCount | 3 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001312587900001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1568-4946 |
| IngestDate | Sat Nov 29 03:06:05 EST 2025 Tue Nov 18 21:44:17 EST 2025 Sat Nov 23 15:54:55 EST 2024 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Keywords | Knowledge graph embedding Multi-modal representation learning Natural language processing Trigger-action programming |
| Language | English |
| LinkModel | OpenURL |
| MergedId | FETCHMERGED-LOGICAL-c300t-71ba3940cb35d07576193b7facc03f948785bc429ccfed1179cd7e5bfd46b55e3 |
| ParticipantIDs | crossref_primary_10_1016_j_asoc_2024_112163 crossref_citationtrail_10_1016_j_asoc_2024_112163 elsevier_sciencedirect_doi_10_1016_j_asoc_2024_112163 |
| PublicationCentury | 2000 |
| PublicationDate | November 2024 2024-11-00 |
| PublicationDateYYYYMMDD | 2024-11-01 |
| PublicationDate_xml | – month: 11 year: 2024 text: November 2024 |
| PublicationDecade | 2020 |
| PublicationTitle | Applied soft computing |
| PublicationYear | 2024 |
| Publisher | Elsevier B.V |
| Publisher_xml | – name: Elsevier B.V |
| References | Reimers, Gurevych (b49) 2019 Corno, De Russis, Monge Roffarello (b3) 2021; 39 X. Yang, M. Yan, S. Pan, X. Ye, D. Fan, Simple and efficient heterogeneous graph neural network, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, No. 9, 2023, pp. 10816–10824. Liu, Li (b45) 2018; 6 Dai, Wang, Xiong, Guo (b46) 2020; 9 Liu, Liu, Jiang, Fan, Luo (b8) 2020; 30 Ni, Huang, Hu, Lin (b13) 2022; 582 K. Wang, R. He, W. Wang, L. Wang, T. Tan, Learning Coupled Feature Spaces for Cross-Modal Matching, in: Proceedings of the IEEE International Conference on Computer Vision, ICCV, 2013. Hu, Wu, Xing, Wang (b24) 2019; 7 Y. Yao, M.M. Kamani, Z. Cheng, L. Chen, C. Joe-Wong, T. Liu, FedRule: Federated Rule Recommendation System with Graph Neural Networks, in: Proceedings of the 8th ACM/IEEE Conference on Internet of Things Design and Implementation, 2023, pp. 197–208. M. Cornia, L. Baraldi, R. Cucchiara, Show, control and tell: A framework for generating controllable and grounded captions, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 8307–8316. H. Caselles-Dupré, F. Lesaint, J. Royo-Letelier, Word2vec applied to recommendation: Hyperparameters matter, in: Proceedings of the 12th ACM Conference on Recommender Systems, 2018, pp. 352–356. Liu, Zhang, Deng, Xie, Liu, Li (b39) 2023 Li, Liu, Zhang, Lin, Fang, Li, Xiong (b23) 2021; 455 Huang, Xu, Ni, Zhu, Wang (b40) 2019; 6 F. Corno, L. De Russis, A. Monge Roffarello, Empowering End Users in Debugging Trigger-Action Rules, in: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19, New York, NY, USA, 2019, pp. 1–13. Castellano, Digeno, Sansaro, Vessio (b34) 2022; 248 W. Brackenbury, A. Deora, J. Ritchey, J. Vallee, W. He, G. Wang, M.L. Littman, B. Ur, How users interpret bugs in trigger-action programming, in: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 2019, pp. 1–12. Zhu, Liu, Liu (b30) 2021; 58 Corno, De Russis, Roffarello (b18) 2017; 50 X. Mi, F. Qian, Y. Zhang, X. Wang, An empirical characterization of IFTTT: ecosystem, usage, and performance, in: Proceedings of the 2017 Internet Measurement Conference, 2017, pp. 398–404. Liu, Li, Zhang, Hao, Ma, Wang (b14) 2024 Kim, Suh, Lee (b25) 2022; 59 Wu, Shen, Van Den Hengel (b43) 2019; 90 Liu, Zhang, Deng, Liu, Zhang, Li (b38) 2023; 32 Mattioli, Paternò (b19) 2021 Wu, Dinkel, Yu (b10) 2019 Yang, Zhang, Xu (b44) 2016; 12 L. Zhou, Y. Zhou, J.J. Corso, R. Socher, C. Xiong, End-to-end dense video captioning with masked transformer, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 8739–8748. Z. Zhao, H. Bai, J. Zhang, Y. Zhang, S. Xu, Z. Lin, R. Timofte, L. Van Gool, Cddfuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 5906–5916. Corno, De Russis, Monge Roffarello (b28) 2019; 10 Q. You, Z. Zhang, J. Luo, End-to-end convolutional semantic embeddings, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 5735–5744. Sun, Deng, Nie, Tang (b48) 2019 I.N.B. Yusuf, L. Jiang, D. Lo, Accurate generation of trigger-action programs with domain-adapted sequence-to-sequence learning, in: Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, 2022, pp. 99–110. C. Quirk, R. Mooney, M. Galley, Language to code: Learning semantic parsers for if-this-then-that recipes, in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2015, pp. 878–888. Wu, Hu, Mao, Xing, Wang (b7) 2024; 235 Yun, Jeong, Yoo, Lee, Sean, Kim, Kang, Kim (b50) 2022; 153 P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, Y. Bengio, Graph Attention Networks, in: International Conference on Learning Representations, 2018. Ricci, Rokach, Shapira (b52) 2011 Hu, Gong, Xing, Wang (b26) 2019; 7 J. Lian, X. Zhou, F. Zhang, Z. Chen, X. Xie, G. Sun, xdeepfm: Combining explicit and implicit feature interactions for recommender systems, in: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018, pp. 1754–1763. Xing, Hu, Zhang, Wu, Wang (b5) 2021; 18 Z. Wang, J. Zhang, J. Feng, Z. Chen, Knowledge graph and text jointly embedding, in: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP, 2014, pp. 1591–1601. Liu, Zheng, Li, Zhang, Lin, Shen, Xiong, Wang (b22) 2022; 468 Chimamiwa, Alirezaie, Pecora, Loutfi (b53) 2021; 34 Thomsen, Giaretta, Dragoni (b29) 2020 Mattioli, Paternò (b20) 2020 C. Zhang, H. Liu, Y. Deng, B. Xie, Y. Li, TokenHPE: Learning orientation tokens for efficient head pose estimation via transformers, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 8897–8906. Liu, Zheng, Li, Shen, Lin, Wang, Zhang, Zhang, Xiong (b21) 2021; 18 Park, Bae, Kim, Kim, Choi (b32) 2022 A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, in: ICLR, 2021. Wu, Duan, Yue, Zhang (b12) 2021; 15 Mikolov, Sutskever, Chen, Corrado, Dean (b16) 2013; 26 Mattioli (10.1016/j.asoc.2024.112163_b20) 2020 Dai (10.1016/j.asoc.2024.112163_b46) 2020; 9 Li (10.1016/j.asoc.2024.112163_b23) 2021; 455 Hu (10.1016/j.asoc.2024.112163_b26) 2019; 7 Thomsen (10.1016/j.asoc.2024.112163_b29) 2020 Reimers (10.1016/j.asoc.2024.112163_b49) 2019 Liu (10.1016/j.asoc.2024.112163_b39) 2023 10.1016/j.asoc.2024.112163_b41 10.1016/j.asoc.2024.112163_b42 Liu (10.1016/j.asoc.2024.112163_b45) 2018; 6 Ricci (10.1016/j.asoc.2024.112163_b52) 2011 10.1016/j.asoc.2024.112163_b47 Corno (10.1016/j.asoc.2024.112163_b18) 2017; 50 Corno (10.1016/j.asoc.2024.112163_b28) 2019; 10 Wu (10.1016/j.asoc.2024.112163_b10) 2019 10.1016/j.asoc.2024.112163_b6 10.1016/j.asoc.2024.112163_b9 Wu (10.1016/j.asoc.2024.112163_b43) 2019; 90 10.1016/j.asoc.2024.112163_b31 10.1016/j.asoc.2024.112163_b1 10.1016/j.asoc.2024.112163_b2 10.1016/j.asoc.2024.112163_b35 10.1016/j.asoc.2024.112163_b4 10.1016/j.asoc.2024.112163_b33 Sun (10.1016/j.asoc.2024.112163_b48) 2019 10.1016/j.asoc.2024.112163_b36 10.1016/j.asoc.2024.112163_b37 Wu (10.1016/j.asoc.2024.112163_b12) 2021; 15 Ni (10.1016/j.asoc.2024.112163_b13) 2022; 582 Park (10.1016/j.asoc.2024.112163_b32) 2022 Wu (10.1016/j.asoc.2024.112163_b7) 2024; 235 Mattioli (10.1016/j.asoc.2024.112163_b19) 2021 Kim (10.1016/j.asoc.2024.112163_b25) 2022; 59 Liu (10.1016/j.asoc.2024.112163_b22) 2022; 468 Hu (10.1016/j.asoc.2024.112163_b24) 2019; 7 Mikolov (10.1016/j.asoc.2024.112163_b16) 2013; 26 10.1016/j.asoc.2024.112163_b27 Xing (10.1016/j.asoc.2024.112163_b5) 2021; 18 Zhu (10.1016/j.asoc.2024.112163_b30) 2021; 58 Yang (10.1016/j.asoc.2024.112163_b44) 2016; 12 Liu (10.1016/j.asoc.2024.112163_b38) 2023; 32 Corno (10.1016/j.asoc.2024.112163_b3) 2021; 39 Liu (10.1016/j.asoc.2024.112163_b14) 2024 10.1016/j.asoc.2024.112163_b51 Liu (10.1016/j.asoc.2024.112163_b8) 2020; 30 Huang (10.1016/j.asoc.2024.112163_b40) 2019; 6 Yun (10.1016/j.asoc.2024.112163_b50) 2022; 153 10.1016/j.asoc.2024.112163_b11 10.1016/j.asoc.2024.112163_b17 Liu (10.1016/j.asoc.2024.112163_b21) 2021; 18 10.1016/j.asoc.2024.112163_b15 Chimamiwa (10.1016/j.asoc.2024.112163_b53) 2021; 34 Castellano (10.1016/j.asoc.2024.112163_b34) 2022; 248 |
| References_xml | – volume: 153 start-page: 104 year: 2022 end-page: 119 ident: b50 article-title: Graph transformer networks: Learning meta-path graphs to improve GNNs publication-title: Neural Netw. – reference: X. Mi, F. Qian, Y. Zhang, X. Wang, An empirical characterization of IFTTT: ecosystem, usage, and performance, in: Proceedings of the 2017 Internet Measurement Conference, 2017, pp. 398–404. – volume: 18 start-page: 1231 year: 2021 end-page: 1239 ident: b5 article-title: Nonnegative matrix factorization based heterogeneous graph embedding method for trigger-action programming in IoT publication-title: IEEE Trans. Ind. Inform. – reference: P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, Y. Bengio, Graph Attention Networks, in: International Conference on Learning Representations, 2018. – volume: 58 year: 2021 ident: b30 article-title: Learning multimodal word representation with graph convolutional networks publication-title: Inf. Process. Manage. – reference: L. Zhou, Y. Zhou, J.J. Corso, R. Socher, C. Xiong, End-to-end dense video captioning with masked transformer, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 8739–8748. – year: 2024 ident: b14 article-title: Cross-modality gesture recognition with complete representation projection publication-title: IEEE Internet Things J. – year: 2019 ident: b48 article-title: RotatE: Knowledge graph embedding by relational rotation in complex space publication-title: International Conference on Learning Representations – volume: 248 year: 2022 ident: b34 article-title: Leveraging knowledge graphs and deep learning for automatic art analysis publication-title: Knowl.-Based Syst. – reference: A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, in: ICLR, 2021. – start-page: 261 year: 2022 end-page: 281 ident: b32 article-title: Graph-text multi-modal pre-training for medical representation learning publication-title: Conference on Health, Inference, and Learning – start-page: 3980 year: 2019 end-page: 3990 ident: b49 article-title: Sentence-BERT: Sentence embeddings using siamese BERT-networks publication-title: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019 – reference: C. Quirk, R. Mooney, M. Galley, Language to code: Learning semantic parsers for if-this-then-that recipes, in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2015, pp. 878–888. – reference: X. Yang, M. Yan, S. Pan, X. Ye, D. Fan, Simple and efficient heterogeneous graph neural network, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, No. 9, 2023, pp. 10816–10824. – start-page: 1 year: 2021 end-page: 11 ident: b19 article-title: Recommendations for creating trigger-action rules in a block-based environment publication-title: Behav. Inf. Technol. – volume: 10 start-page: 1 year: 2019 end-page: 27 ident: b28 article-title: RecRules: recommending IF-THEN rules for end-user development publication-title: ACM Trans. Intell. Syst. Technol. – volume: 15 start-page: 3330 year: 2021 end-page: 3343 ident: b12 article-title: Mashup-oriented web API recommendation via multi-model fusion and multi-task learning publication-title: IEEE Trans. Serv. Comput. – year: 2023 ident: b39 article-title: Transifc: invariant cues-aware feature concentration learning for efficient fine-grained bird image classification publication-title: IEEE Trans. Multimed. – volume: 6 start-page: 10675 year: 2019 end-page: 10685 ident: b40 article-title: Multimodal representation learning for recommendation in internet of things publication-title: IEEE Internet Things J. – reference: W. Brackenbury, A. Deora, J. Ritchey, J. Vallee, W. He, G. Wang, M.L. Littman, B. Ur, How users interpret bugs in trigger-action programming, in: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 2019, pp. 1–12. – start-page: 830 year: 2019 end-page: 834 ident: b10 article-title: Audio caption: Listen and tell publication-title: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing – reference: Z. Wang, J. Zhang, J. Feng, Z. Chen, Knowledge graph and text jointly embedding, in: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP, 2014, pp. 1591–1601. – volume: 18 start-page: 4361 year: 2021 end-page: 4371 ident: b21 article-title: EDMF: Efficient deep matrix factorization with review feature learning for industrial recommender system publication-title: IEEE Trans. Ind. Inform. – volume: 6 start-page: 6034 year: 2018 end-page: 6041 ident: b45 article-title: Multimodal GAN for energy efficiency and cloud classification in internet of things publication-title: IEEE Internet Things J. – volume: 39 start-page: 1 year: 2021 end-page: 33 ident: b3 article-title: From users’ intentions to if-then rules in the internet of things publication-title: ACM Trans. Inf. Syst. (TOIS) – volume: 12 start-page: 1 year: 2016 end-page: 22 ident: b44 article-title: Semantic feature mining for video event understanding publication-title: ACM Trans. Multimedia Comput. Commun. Appl. (TOMM) – volume: 32 start-page: 6289 year: 2023 end-page: 6302 ident: b38 article-title: Orientation cues-aware facial relationship representation for head pose estimation via transformer publication-title: IEEE Trans. Image Process. – start-page: 1 year: 2011 end-page: 35 ident: b52 article-title: Introduction to recommender systems handbook publication-title: Recommender Systems Handbook – volume: 59 year: 2022 ident: b25 article-title: What IoT devices and applications should be connected? Predicting user behaviors of IoT services with node2vec embedding publication-title: Inf. Process. Manage. – year: 2020 ident: b20 article-title: A visual environment for end-user creation of IoT customization rules with recommendation support – volume: 30 start-page: 1261 year: 2020 end-page: 1274 ident: b8 article-title: A bilevel integrated model with data-driven layer ensemble for multi-modality image fusion publication-title: IEEE Trans. Image Process. – reference: I.N.B. Yusuf, L. Jiang, D. Lo, Accurate generation of trigger-action programs with domain-adapted sequence-to-sequence learning, in: Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, 2022, pp. 99–110. – volume: 582 start-page: 22 year: 2022 end-page: 37 ident: b13 article-title: A two-stage embedding model for recommendation with multimodal auxiliary information publication-title: Inform. Sci. – volume: 468 start-page: 469 year: 2022 end-page: 481 ident: b22 article-title: Multi-perspective social recommendation method with graph representation learning publication-title: Neurocomputing – volume: 7 start-page: 1939 year: 2019 end-page: 1948 ident: b24 article-title: Things2Vec: Semantic modeling in the internet of things with graph representation learning publication-title: IEEE Internet Things J. – reference: C. Zhang, H. Liu, Y. Deng, B. Xie, Y. Li, TokenHPE: Learning orientation tokens for efficient head pose estimation via transformers, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 8897–8906. – reference: M. Cornia, L. Baraldi, R. Cucchiara, Show, control and tell: A framework for generating controllable and grounded captions, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 8307–8316. – volume: 9 start-page: 750 year: 2020 ident: b46 article-title: A survey on knowledge graph embedding: Approaches, applications and benchmarks publication-title: Electronics – reference: K. Wang, R. He, W. Wang, L. Wang, T. Tan, Learning Coupled Feature Spaces for Cross-Modal Matching, in: Proceedings of the IEEE International Conference on Computer Vision, ICCV, 2013. – volume: 235 year: 2024 ident: b7 article-title: A data fusion framework based on heterogeneous information network embedding for trigger-action programming in IoT publication-title: Expert Syst. Appl. – volume: 50 start-page: 18 year: 2017 end-page: 24 ident: b18 article-title: A semantic web approach to simplifying trigger-action programming in the IoT publication-title: Computer – volume: 455 start-page: 283 year: 2021 end-page: 296 ident: b23 article-title: CARM: Confidence-aware recommender model via review representation learning and historical rating behavior in the online platforms publication-title: Neurocomputing – reference: Z. Zhao, H. Bai, J. Zhang, Y. Zhang, S. Xu, Z. Lin, R. Timofte, L. Van Gool, Cddfuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 5906–5916. – volume: 26 year: 2013 ident: b16 article-title: Distributed representations of words and phrases and their compositionality publication-title: Adv. Neural Inf. Process. Syst. – volume: 7 start-page: 31233 year: 2019 end-page: 31242 ident: b26 article-title: Semantic representation with heterogeneous information network using matrix factorization for clustering in the internet of things publication-title: IEEE Access – reference: J. Lian, X. Zhou, F. Zhang, Z. Chen, X. Xie, G. Sun, xdeepfm: Combining explicit and implicit feature interactions for recommender systems, in: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018, pp. 1754–1763. – volume: 34 year: 2021 ident: b53 article-title: Multi-sensor dataset of human activities in a smart home environment publication-title: Data Brief – reference: F. Corno, L. De Russis, A. Monge Roffarello, Empowering End Users in Debugging Trigger-Action Rules, in: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19, New York, NY, USA, 2019, pp. 1–13. – reference: H. Caselles-Dupré, F. Lesaint, J. Royo-Letelier, Word2vec applied to recommendation: Hyperparameters matter, in: Proceedings of the 12th ACM Conference on Recommender Systems, 2018, pp. 352–356. – volume: 90 start-page: 119 year: 2019 end-page: 133 ident: b43 article-title: Wider or deeper: Revisiting the resnet model for visual recognition publication-title: Pattern Recognit. – start-page: 85 year: 2020 end-page: 99 ident: b29 article-title: Smart lamp or security camera? Automatic identification of IoT devices publication-title: International Networking Conference – reference: Y. Yao, M.M. Kamani, Z. Cheng, L. Chen, C. Joe-Wong, T. Liu, FedRule: Federated Rule Recommendation System with Graph Neural Networks, in: Proceedings of the 8th ACM/IEEE Conference on Internet of Things Design and Implementation, 2023, pp. 197–208. – reference: Q. You, Z. Zhang, J. Luo, End-to-end convolutional semantic embeddings, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 5735–5744. – year: 2020 ident: 10.1016/j.asoc.2024.112163_b20 – ident: 10.1016/j.asoc.2024.112163_b15 doi: 10.1109/CVPR.2018.00601 – volume: 58 issue: 6 year: 2021 ident: 10.1016/j.asoc.2024.112163_b30 article-title: Learning multimodal word representation with graph convolutional networks publication-title: Inf. Process. Manage. doi: 10.1016/j.ipm.2021.102709 – ident: 10.1016/j.asoc.2024.112163_b6 doi: 10.1145/3576842.3582328 – volume: 10 start-page: 1 issue: 5 year: 2019 ident: 10.1016/j.asoc.2024.112163_b28 article-title: RecRules: recommending IF-THEN rules for end-user development publication-title: ACM Trans. Intell. Syst. Technol. doi: 10.1145/3344211 – ident: 10.1016/j.asoc.2024.112163_b35 – ident: 10.1016/j.asoc.2024.112163_b4 doi: 10.1145/3524610.3527922 – ident: 10.1016/j.asoc.2024.112163_b33 doi: 10.3115/v1/D14-1167 – volume: 26 year: 2013 ident: 10.1016/j.asoc.2024.112163_b16 article-title: Distributed representations of words and phrases and their compositionality publication-title: Adv. Neural Inf. Process. Syst. – volume: 468 start-page: 469 year: 2022 ident: 10.1016/j.asoc.2024.112163_b22 article-title: Multi-perspective social recommendation method with graph representation learning publication-title: Neurocomputing doi: 10.1016/j.neucom.2021.10.050 – ident: 10.1016/j.asoc.2024.112163_b47 doi: 10.1145/3131365.3131369 – volume: 90 start-page: 119 year: 2019 ident: 10.1016/j.asoc.2024.112163_b43 article-title: Wider or deeper: Revisiting the resnet model for visual recognition publication-title: Pattern Recognit. doi: 10.1016/j.patcog.2019.01.006 – ident: 10.1016/j.asoc.2024.112163_b41 doi: 10.1145/3219819.3220023 – volume: 235 year: 2024 ident: 10.1016/j.asoc.2024.112163_b7 article-title: A data fusion framework based on heterogeneous information network embedding for trigger-action programming in IoT publication-title: Expert Syst. Appl. doi: 10.1016/j.eswa.2023.121065 – year: 2024 ident: 10.1016/j.asoc.2024.112163_b14 article-title: Cross-modality gesture recognition with complete representation projection publication-title: IEEE Internet Things J. – ident: 10.1016/j.asoc.2024.112163_b37 doi: 10.1109/CVPR52729.2023.00859 – volume: 18 start-page: 1231 issue: 2 year: 2021 ident: 10.1016/j.asoc.2024.112163_b5 article-title: Nonnegative matrix factorization based heterogeneous graph embedding method for trigger-action programming in IoT publication-title: IEEE Trans. Ind. Inform. doi: 10.1109/TII.2021.3092774 – start-page: 3980 year: 2019 ident: 10.1016/j.asoc.2024.112163_b49 article-title: Sentence-BERT: Sentence embeddings using siamese BERT-networks – volume: 7 start-page: 1939 issue: 3 year: 2019 ident: 10.1016/j.asoc.2024.112163_b24 article-title: Things2Vec: Semantic modeling in the internet of things with graph representation learning publication-title: IEEE Internet Things J. doi: 10.1109/JIOT.2019.2962630 – start-page: 830 year: 2019 ident: 10.1016/j.asoc.2024.112163_b10 article-title: Audio caption: Listen and tell – volume: 59 issue: 2 year: 2022 ident: 10.1016/j.asoc.2024.112163_b25 article-title: What IoT devices and applications should be connected? Predicting user behaviors of IoT services with node2vec embedding publication-title: Inf. Process. Manage. doi: 10.1016/j.ipm.2022.102869 – volume: 18 start-page: 4361 issue: 7 year: 2021 ident: 10.1016/j.asoc.2024.112163_b21 article-title: EDMF: Efficient deep matrix factorization with review feature learning for industrial recommender system publication-title: IEEE Trans. Ind. Inform. doi: 10.1109/TII.2021.3128240 – year: 2019 ident: 10.1016/j.asoc.2024.112163_b48 article-title: RotatE: Knowledge graph embedding by relational rotation in complex space – ident: 10.1016/j.asoc.2024.112163_b1 doi: 10.1145/3290605.3300618 – start-page: 1 year: 2021 ident: 10.1016/j.asoc.2024.112163_b19 article-title: Recommendations for creating trigger-action rules in a block-based environment publication-title: Behav. Inf. Technol. – ident: 10.1016/j.asoc.2024.112163_b17 doi: 10.1145/3240323.3240377 – volume: 9 start-page: 750 issue: 5 year: 2020 ident: 10.1016/j.asoc.2024.112163_b46 article-title: A survey on knowledge graph embedding: Approaches, applications and benchmarks publication-title: Electronics doi: 10.3390/electronics9050750 – year: 2023 ident: 10.1016/j.asoc.2024.112163_b39 article-title: Transifc: invariant cues-aware feature concentration learning for efficient fine-grained bird image classification publication-title: IEEE Trans. Multimed. – volume: 32 start-page: 6289 year: 2023 ident: 10.1016/j.asoc.2024.112163_b38 article-title: Orientation cues-aware facial relationship representation for head pose estimation via transformer publication-title: IEEE Trans. Image Process. doi: 10.1109/TIP.2023.3331309 – volume: 455 start-page: 283 year: 2021 ident: 10.1016/j.asoc.2024.112163_b23 article-title: CARM: Confidence-aware recommender model via review representation learning and historical rating behavior in the online platforms publication-title: Neurocomputing doi: 10.1016/j.neucom.2021.03.122 – volume: 39 start-page: 1 issue: 4 year: 2021 ident: 10.1016/j.asoc.2024.112163_b3 article-title: From users’ intentions to if-then rules in the internet of things publication-title: ACM Trans. Inf. Syst. (TOIS) doi: 10.1145/3447264 – volume: 50 start-page: 18 issue: 11 year: 2017 ident: 10.1016/j.asoc.2024.112163_b18 article-title: A semantic web approach to simplifying trigger-action programming in the IoT publication-title: Computer doi: 10.1109/MC.2017.4041355 – volume: 248 year: 2022 ident: 10.1016/j.asoc.2024.112163_b34 article-title: Leveraging knowledge graphs and deep learning for automatic art analysis publication-title: Knowl.-Based Syst. doi: 10.1016/j.knosys.2022.108859 – start-page: 1 year: 2011 ident: 10.1016/j.asoc.2024.112163_b52 article-title: Introduction to recommender systems handbook – ident: 10.1016/j.asoc.2024.112163_b51 doi: 10.1609/aaai.v37i9.26283 – volume: 582 start-page: 22 year: 2022 ident: 10.1016/j.asoc.2024.112163_b13 article-title: A two-stage embedding model for recommendation with multimodal auxiliary information publication-title: Inform. Sci. doi: 10.1016/j.ins.2021.09.006 – ident: 10.1016/j.asoc.2024.112163_b31 doi: 10.1109/ICCV.2013.261 – start-page: 85 year: 2020 ident: 10.1016/j.asoc.2024.112163_b29 article-title: Smart lamp or security camera? Automatic identification of IoT devices – volume: 30 start-page: 1261 year: 2020 ident: 10.1016/j.asoc.2024.112163_b8 article-title: A bilevel integrated model with data-driven layer ensemble for multi-modality image fusion publication-title: IEEE Trans. Image Process. doi: 10.1109/TIP.2020.3043125 – ident: 10.1016/j.asoc.2024.112163_b36 – ident: 10.1016/j.asoc.2024.112163_b9 doi: 10.1109/CVPR52729.2023.00572 – volume: 15 start-page: 3330 issue: 6 year: 2021 ident: 10.1016/j.asoc.2024.112163_b12 article-title: Mashup-oriented web API recommendation via multi-model fusion and multi-task learning publication-title: IEEE Trans. Serv. Comput. doi: 10.1109/TSC.2021.3098756 – volume: 153 start-page: 104 year: 2022 ident: 10.1016/j.asoc.2024.112163_b50 article-title: Graph transformer networks: Learning meta-path graphs to improve GNNs publication-title: Neural Netw. doi: 10.1016/j.neunet.2022.05.026 – start-page: 261 year: 2022 ident: 10.1016/j.asoc.2024.112163_b32 article-title: Graph-text multi-modal pre-training for medical representation learning – ident: 10.1016/j.asoc.2024.112163_b42 doi: 10.1109/CVPR.2018.00911 – volume: 12 start-page: 1 issue: 4 year: 2016 ident: 10.1016/j.asoc.2024.112163_b44 article-title: Semantic feature mining for video event understanding publication-title: ACM Trans. Multimedia Comput. Commun. Appl. (TOMM) doi: 10.1145/2962719 – ident: 10.1016/j.asoc.2024.112163_b27 doi: 10.3115/v1/P15-1085 – ident: 10.1016/j.asoc.2024.112163_b11 doi: 10.1109/CVPR.2019.00850 – volume: 34 year: 2021 ident: 10.1016/j.asoc.2024.112163_b53 article-title: Multi-sensor dataset of human activities in a smart home environment publication-title: Data Brief doi: 10.1016/j.dib.2020.106632 – volume: 6 start-page: 10675 issue: 6 year: 2019 ident: 10.1016/j.asoc.2024.112163_b40 article-title: Multimodal representation learning for recommendation in internet of things publication-title: IEEE Internet Things J. doi: 10.1109/JIOT.2019.2940709 – ident: 10.1016/j.asoc.2024.112163_b2 doi: 10.1145/3290605.3300782 – volume: 6 start-page: 6034 issue: 4 year: 2018 ident: 10.1016/j.asoc.2024.112163_b45 article-title: Multimodal GAN for energy efficiency and cloud classification in internet of things publication-title: IEEE Internet Things J. doi: 10.1109/JIOT.2018.2866328 – volume: 7 start-page: 31233 year: 2019 ident: 10.1016/j.asoc.2024.112163_b26 article-title: Semantic representation with heterogeneous information network using matrix factorization for clustering in the internet of things publication-title: IEEE Access doi: 10.1109/ACCESS.2019.2903310 |
| SSID | ssj0016928 |
| Score | 2.4394805 |
| Snippet | The escalating popularity of smart devices has given rise to an increasing trend wherein users leverage customized trigger-action programming (TAP) rules... |
| SourceID | crossref elsevier |
| SourceType | Enrichment Source Index Database Publisher |
| StartPage | 112163 |
| SubjectTerms | Knowledge graph embedding Multi-modal representation learning Natural language processing Trigger-action programming |
| Title | TAP with ease: a generic recommendation system for trigger-action programming based on multi-modal representation learning |
| URI | https://dx.doi.org/10.1016/j.asoc.2024.112163 |
| Volume | 166 |
| WOSCitedRecordID | wos001312587900001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVESC databaseName: Elsevier SD Freedom Collection Journals 2021 issn: 1568-4946 databaseCode: AIEXJ dateStart: 20010601 customDbUrl: isFulltext: true dateEnd: 99991231 titleUrlDefault: https://www.sciencedirect.com omitProxy: false ssIdentifier: ssj0016928 providerName: Elsevier |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtV3Na9swFBdtukMv27qtNN1WdOhtODiWZVu7hdGxFRpySCE3oy-HlsYJ-Shlf32fLMlxmi0sh15MEPKz8fvl6aen94HQZUgjmYU8DUIViSBmXAVMhCoAppDwSMACTmyzibTfz0YjNnA-3UXVTiAty-zpic1eVdUwBso2qbN7qLsWCgPwG5QOV1A7XP9P8b2B9a6agxebyzw2taWrWs3wwIl2fZRcFWcbaAibdOPec53DXdTWxPgRzDqnzJlCFXsYTKaq6gUwW-ctlb73xLhJdT2_XYChryLXV0s_wywCq8ojzxsjznN9c7c9Bh943PRPRLFL1GuY1CQDEDhHo7e5SdNqAufrWjO3ZdCtb-G-wwGrHSO-s568WT37xapWxxr6MLb73MjIjYzcyjhER1FKWdZCR73fV6Pr-vQpYVVP3vrNXbKVjQt8-SZ_JzQNkjJ8j9663QXuWVScoANdfkDvfOcO7Az5R_QHQIINSLAByXfMsYMI3oQIthDBABG8CRHcgAiuIIJhsAERvAkR7CHyCd3-vBr--BW4LhyBJGG4DNKu4ITFoRSEKiCYxu9FRFpwKUNSMNjwZlRIoDVSFlqZCoNSpZqKQsWJoFSTU9Qqp6U-QziShFGuY80IEHcSCRDHBaNZRguukm4bdf2nzKUrUW86pTzk_1ZiG32r75nZAi07Z1OvodxRTEsdcwDcjvvO93rKZ3S8_id8Qa3lfKW_ojfycXm3mF84tD0DMCafng |
| linkProvider | Elsevier |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=TAP+with+ease%3A+a+generic+recommendation+system+for+trigger-action+programming+based+on+multi-modal+representation+learning&rft.jtitle=Applied+soft+computing&rft.au=Wu%2C+Gang&rft.au=Wang%2C+Ming&rft.au=Wang%2C+Feng&rft.date=2024-11-01&rft.issn=1568-4946&rft.volume=166&rft.spage=112163&rft_id=info:doi/10.1016%2Fj.asoc.2024.112163&rft.externalDBID=n%2Fa&rft.externalDocID=10_1016_j_asoc_2024_112163 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1568-4946&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1568-4946&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1568-4946&client=summon |