In EDS ansehen

English-focused CL-HAMC with contrastive learning and hierarchical attention for multiple-choice reading comprehension.

Gespeichert in:

Bibliographische Detailangaben
Titel:	English-focused CL-HAMC with contrastive learning and hierarchical attention for multiple-choice reading comprehension.
Autoren:	Ji L; Xinyang Agriculture and Forestry University, Xinyang, 464000, China., Yao L; Xinyang Agriculture and Forestry University, Xinyang, 464000, China. 20180100041@csuft.edu.cn., Xu W; Hangzhou Allsheng Instruments Co., Ltd., Hangzhou, 310024, China.
Quelle:	Scientific reports [Sci Rep] 2025 Nov 17; Vol. 15 (1), pp. 40246. Date of Electronic Publication: 2025 Nov 17.
Publikationsart:	Journal Article
Sprache:	English
Info zur Zeitschrift:	Publisher: Nature Publishing Group Country of Publication: England NLM ID: 101563288 Publication Model: Electronic Cited Medium: Internet ISSN: 2045-2322 (Electronic) Linking ISSN: 20452322 NLM ISO Abbreviation: Sci Rep Subsets: MEDLINE
Imprint Name(s):	Original Publication: London : Nature Publishing Group, copyright 2011-
MeSH-Schlagworte:	Comprehension* , Reading* , Language* , Learning* , Attention*, Humans ; Semantics ; Artificial Intelligence
Abstract:	Competing Interests: Declarations. Competing interests: The authors declare no competing interests. Multiple-choice questions constitute a critical format for assessing language application proficiency in standardized English tests, such as BEC and TOEIC. Developing explanatory content for such materials traditionally relies heavily on manual labor in test item analysis, which is labor-intensive and time-consuming. Consequently, Artificial Intelligence (AI) approaches centered on Machine Reading Comprehension for Multiple Choice (MCRC) are becoming the preferred solution for generating auxiliary educational content. This task demands models capable of profoundly understanding textual semantics and accurately identifying complex relationship patterns between passages, questions, and answer options. Although Pre-trained Language Models (PLMs) have achieved remarkable success on MCRC tasks, existing methods confront two primary limitations: (1) they remain susceptible to misclassifying highly textually similar yet semantically distant distractor options (e.g. synonymous business terms); (2) they exhibit significantly diminished accuracy when tackling questions requiring indirect reasoning or background knowledge to identify implicit answers. To address these challenges, this paper proposes Contrastive Learning-driven Hierarchical Attention Model for Multiple Choice (CL-HAMC). The proposed model innovatively employs multi-head attention mechanisms to hierarchically model the triple interactions among passages, questions, and options, simulating the progressive, multi-layered reasoning process humans undertake during problem-solving. Furthermore, it incorporates a contrastive learning strategy to sharpen the model's ability to discern nuanced semantic distinctions among answer choices. Extensive experiments on the RACE, RACE-M, and RACE-H benchmarks demonstrate that CL-HAMC achieves substantial and consistent performance gains, establishing a new state-of-the-art (SOTA) on all three datasets. Moreover, CL-HAMC exhibits competitive results on the DREAM dataset. This study provides an effective solution towards the automated processing of highly distractor-rich multiple-choice questions within the English auxiliary learning domain. (© 2025. The Author(s).)
References:	Hermann, K. M. et al. Teaching machines to read and comprehend. In Proc. of the 29th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15, 1693–1701 (MIT Press, 2015). Lai, G., Xie, Q., Liu, H., Yang, Y. & Hovy, E. RACE: Large-scale ReAding comprehension dataset from examinations. In Proc. of the 2017 Conference on Empirical Methods in Natural Language Processing (eds. Palmer, M. et al.) 785–794 (Association for Computational Linguistics, 2017), https://doi.org/10.18653/v1/D17-1082 . Rajpurkar, P., Jia, R. & Liang, P. Know what you don’t know: Unanswerable questions for SQuAD. In Proc. of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (eds. Gurevych, I. & Miyao, Y.) 784–789 (Association for Computational Linguistics, 2018), https://doi.org/10.18653/v1/P18-2124 . Nguyen, T. et al. Ms marco: A human generated machine reading comprehension dataset. Choice 2640, 660 (2016). Bridge, M. Cif and fob contracts in English law: Current issues and problems. In Research Handbook on International and Comparative Sale of Goods Law (ed. Bridge, M.) 213–239 (Edward Elgar Publishing, 2019). Jung, Y. Korean Business Communication: A Comprehensive Introduction (Taylor & Francis, 2022). (PMID: 10.4324/9781003108061) Sun, K. et al. DREAM: A challenge data set and models for dialogue-based reading comprehension. Tran. Assoc. Comput. Linguist. 7, 217–231. https://doi.org/10.1162/tacl_a_00264 (2019). (PMID: 10.1162/tacl_a_00264) Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proce. of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (eds. Burstein, J. et al.) 4171–4186, https://doi.org/10.18653/v1/N19-1423 (Association for Computational Linguistics, 2019). Zhuang, L., Wayne, L., Ya, S. & Jun, Z. A robustly optimized BERT pre-training approach with post-training. In Proc. of the 20th Chinese National Conference on Computational Linguistics (eds. Li, S. et al.) 1218–1227 (Chinese Information Processing Society of China, 2021). Lan, Z. et al. Albert: A lite bert for self-supervised learning of language representations. In 8th International Conference on Learning Representations (2020). Zhang, S. et al. Dcmn+: Dual co-matching network for multi-choice reading comprehension. Proc. AAAI Conference on Artificial Intelligence 34, 9563–9570. https://doi.org/10.1609/aaai.v34i05.6502 (2020). (PMID: 10.1609/aaai.v34i05.6502) Jin, D., Gao, S., Kao, J.-Y., Chung, T. & Hakkani-tur, D. Mmm: Multi-stage multi-task learning for multi-choice reading comprehension. Proc. AAAI Conference on Artificial IntelligenceProc. AAAI Conference on Artificial IntelligenceProc. AAAI Conference on Artificial Intelligence 34, 8010–8017. https://doi.org/10.1609/aaai.v34i05.6310 (2020). (PMID: 10.1609/aaai.v34i05.6310) Vaswani, A. et al. Attention is all you need. In Proc. of the 31st International Conference on Neural Information Processing Systems, NIPS’17, 6000–6010 (Curran Associates Inc., 2017). Bahdanau, D., Cho, K. H. & Bengio, Y. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015 (2015). Wang, S., Yu, M., Jiang, J. & Chang, S. A co-matching model for multi-choice reading comprehension. In Proc. of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)(eds. Gurevych, I. & Miyao, Y.) 746–751 (Association for Computational Linguistics, 2018), https://doi.org/10.18653/v1/P18-2118 . Kim, H. & Fung, P. Learning to classify the wrong answers for multiple choice question answering (student abstract). Proc. AAAI Conference on Artificial Intelligence 34, 13843–13844. https://doi.org/10.1609/aaai.v34i10.7194 (2020). (PMID: 10.1609/aaai.v34i10.7194) Zhu, P., Zhang, Z., Zhao, H. & Li, X. Duma: Reading comprehension with transposition thinking. IEEE/ACM Trans. Audio Speech Lang. Proc. 30, 269–279. https://doi.org/10.1109/TASLP.2021.3138683 (2022). (PMID: 10.1109/TASLP.2021.3138683) Longpre, S. et al. A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. In Proc. of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (eds. Duh, K., Gomez, H. & Bethard, S.) 3245–3276 (Association for Computational Linguistics, 2024), https://doi.org/10.18653/v1/2024.naacl-long.179 . Li, J. et al. Datacomp-lm: In search of the next generation of training sets for language models. Adv. Neural Inf. Process. Syst. 37, 14200–14282 (2024). Bai, T. et al. Efficient pretraining data selection for language models via multi-actor collaboration. In Proc. of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (eds. Che, W. et al. 9465–9491 (Association for Computational, 2025), DOI: https://doi.org/10.18653/v1/2025.acl-long.466. Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. A simple framework for contrastive learning of visual representations. In Proc. of the 37th International Conference on Machine Learning, ICML’20 (JMLR.org, 2020). Gao, T., Yao, X. & Chen, D. SimCSE: Simple contrastive learning of sentence embeddings. In Proc. of the 2021 Conference on Empirical Methods in Natural Language Processing (eds. Moens, M.-F. et al.) 6894–6910 (Association for Computational Linguistics, 2021), https://doi.org/10.18653/v1/2021.emnlp-main.552 . Zhang, Y. et al. A contrastive framework for learning sentence representations from pairwise and triple-wise perspective in angular space. In Proc. of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (eds. Muresan, S. et al.) 4892–4903, https://doi.org/10.18653/v1/2022.acl-long.336 (Association for Computational Linguistics, Dublin, Ireland, 2022). Li, S., Hu, X., Lin, L. & Wen, L. Pair-level supervised contrastive learning for natural language inference. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 8237–8241, https://doi.org/10.1109/ICASSP43922.2022.9746499 (2022). Shorinwa, O., Mei, Z., Lidard, J., Ren, A. Z. & Majumdar, A. A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions. ACM Comput. Surv. https://doi.org/10.1145/3744238 (2025). (PMID: 10.1145/3744238) You, Y. et al. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations (2020). Qu, F., Sun, H. & Wu, Y. Unsupervised distractor generation via large language model distilling and counterfactual contrastive decoding. In Ku, L.-W., Martins, A. & Srikumar, V. (eds.) Findings of the Association for Computational Linguistics: ACL 2024, 827–838, https://doi.org/10.18653/v1/2024.findings-acl.47 (Association for Computational Linguistics, Bangkok, Thailand, 2024). Richardson, M., Burges, C. J. & Renshaw, E. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proc. of the 2013 Conference on Empirical Methods in Natural Language Processing (eds. Yarowsky, D. et al.) 193–203 (Association for Computational Linguistics, 2013). Dhingra, B., Liu, H., Yang, Z., Cohen, W. & Salakhutdinov, R. Gated-attention readers for text comprehension. In Proc. of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (eds. Barzilay, R. & Kan, M.-Y.) 1832–1846 (Association for Computational Linguistics, 2017), https://doi.org/10.18653/v1/P17-1168 . Chen, D., Bolton, J. & Manning, C. D. A thorough examination of the CNN/Daily Mail reading comprehension task. In Proc. of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (eds. Erk, K. & Smith, N. A.) 2358–2367 (Association for Computational Linguistics, 2016), https://doi.org/10.18653/v1/P16-1223 . Parikh, S., Sai, A., Nema, P. & Khapra, M. Eliminet: A model for eliminating options for reading comprehension with multiple choice questions. In Proc. of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, 4272–4278 (International Joint Conferences on Artificial Intelligence Organization, 2018), https://doi.org/10.24963/ijcai.2018/594 . Zhu, H., Wei, F., Qin, B. & Liu, T. Hierarchical attention flow for multiple-choice reading comprehension. In Proc. of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’18/IAAI’18/EAAI’18 (AAAI Press, 2018). Radford, A. Improving language understanding with unsupervised learning. OpenAI Res. (2018). Sun, K., Yu, D., Yu, D. & Cardie, C. Improving machine reading comprehension with general reading strategies. In Proc. of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (eds. Burstein, J. et al.) 2633–2643 (Association for Computational Linguistics, 2019), https://doi.org/10.18653/v1/N19-1270 . Yang, Z. et al. XLNet: Generalized Autoregressive Pretraining for Language Understanding (Curran Associates Inc., 2019). He, P., Gao, J. & Chen, W. DeBERTav3: Improving deBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. In The Eleventh International Conference on Learning Representations (2023).
Grant Information:	2024XWH225 Research Project of Henan Province's Cultural Promotion Project in 2024; 2024XWH225 Research Project of Henan Province's Cultural Promotion Project in 2024
Contributed Indexing:	Keywords: Contrastive learning; Deep learning; English; Hierarchical attention; Machine reading comprehension; Multiple-choice QA
Entry Date(s):	Date Created: 20251117 Date Completed: 20251117 Latest Revision: 20251120
Update Code:	20251121
PubMed Central ID:	PMC12623859
DOI:	10.1038/s41598-025-24031-6
PMID:	41249341
Datenbank:	MEDLINE

Full Text Finder

Nájsť tento článok vo Web of Science

Beschreibung
Abstract:	Competing Interests: Declarations. Competing interests: The authors declare no competing interests.<br />Multiple-choice questions constitute a critical format for assessing language application proficiency in standardized English tests, such as BEC and TOEIC. Developing explanatory content for such materials traditionally relies heavily on manual labor in test item analysis, which is labor-intensive and time-consuming. Consequently, Artificial Intelligence (AI) approaches centered on Machine Reading Comprehension for Multiple Choice (MCRC) are becoming the preferred solution for generating auxiliary educational content. This task demands models capable of profoundly understanding textual semantics and accurately identifying complex relationship patterns between passages, questions, and answer options. Although Pre-trained Language Models (PLMs) have achieved remarkable success on MCRC tasks, existing methods confront two primary limitations: (1) they remain susceptible to misclassifying highly textually similar yet semantically distant distractor options (e.g. synonymous business terms); (2) they exhibit significantly diminished accuracy when tackling questions requiring indirect reasoning or background knowledge to identify implicit answers. To address these challenges, this paper proposes Contrastive Learning-driven Hierarchical Attention Model for Multiple Choice (CL-HAMC). The proposed model innovatively employs multi-head attention mechanisms to hierarchically model the triple interactions among passages, questions, and options, simulating the progressive, multi-layered reasoning process humans undertake during problem-solving. Furthermore, it incorporates a contrastive learning strategy to sharpen the model's ability to discern nuanced semantic distinctions among answer choices. Extensive experiments on the RACE, RACE-M, and RACE-H benchmarks demonstrate that CL-HAMC achieves substantial and consistent performance gains, establishing a new state-of-the-art (SOTA) on all three datasets. Moreover, CL-HAMC exhibits competitive results on the DREAM dataset. This study provides an effective solution towards the automated processing of highly distractor-rich multiple-choice questions within the English auxiliary learning domain.<br /> (© 2025. The Author(s).)
ISSN:	2045-2322
DOI:	10.1038/s41598-025-24031-6