Evaluating Terminology Translation in Machine Translation Systems via Metamorphic Testing
Machine translation has become an integral part of daily life, with terminology translation playing a crucial role in ensuring the accuracy of translation results. However, existing translation systems, such as Google Translate, have been shown to occasionally produce errors in terminology translati...
Uloženo v:
| Vydáno v: | IEEE/ACM International Conference on Automated Software Engineering : [proceedings] s. 758 - 769 |
|---|---|
| Hlavní autoři: | , , , |
| Médium: | Konferenční příspěvek |
| Jazyk: | angličtina |
| Vydáno: |
ACM
27.10.2024
|
| Témata: | |
| ISSN: | 2643-1572 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | Machine translation has become an integral part of daily life, with terminology translation playing a crucial role in ensuring the accuracy of translation results. However, existing translation systems, such as Google Translate, have been shown to occasionally produce errors in terminology translation. Current metrics for assessing terminology translation rely on reference translations and bilingual dictionaries, limiting their effectiveness in large-scale automated MT system testing.To address this challenge, we propose a novel method: Metamorphic Testing for Terminology Translation (TermMT), which achieves effective and efficient testing for terminology translation in MT systems without relying on reference translations or bilingual terminology dictionaries. Our approach involves constructing metamorphic relations based on the characteristics of terms: (a) adding an appropriate reference of the term in the given context would not change the translation of the term; (b) if we modify part of a multi-word term, the translation of the revised word combination would change. To evaluate the effectiveness of TermMT, we tested the terminology translation capabilities of three machine translation systems, Google Translate, Bing Microsoft Translator, and mBART, using the English portion of the bilingual UM-corpus dataset. The results show that TermMT detected a total of 3,765 translation errors on Google Translate, 2,351 on Bing Microsoft Translator, and 6,011 on mBART, with precisions of 82.33%, 83.00%, and 86.33%, respectively. |
|---|---|
| AbstractList | Machine translation has become an integral part of daily life, with terminology translation playing a crucial role in ensuring the accuracy of translation results. However, existing translation systems, such as Google Translate, have been shown to occasionally produce errors in terminology translation. Current metrics for assessing terminology translation rely on reference translations and bilingual dictionaries, limiting their effectiveness in large-scale automated MT system testing.To address this challenge, we propose a novel method: Metamorphic Testing for Terminology Translation (TermMT), which achieves effective and efficient testing for terminology translation in MT systems without relying on reference translations or bilingual terminology dictionaries. Our approach involves constructing metamorphic relations based on the characteristics of terms: (a) adding an appropriate reference of the term in the given context would not change the translation of the term; (b) if we modify part of a multi-word term, the translation of the revised word combination would change. To evaluate the effectiveness of TermMT, we tested the terminology translation capabilities of three machine translation systems, Google Translate, Bing Microsoft Translator, and mBART, using the English portion of the bilingual UM-corpus dataset. The results show that TermMT detected a total of 3,765 translation errors on Google Translate, 2,351 on Bing Microsoft Translator, and 6,011 on mBART, with precisions of 82.33%, 83.00%, and 86.33%, respectively. |
| Author | Li, Yanhui Zhang, Xiaofang Wang, Jun Xu, Yihui |
| Author_xml | – sequence: 1 givenname: Yihui surname: Xu fullname: Xu, Yihui email: 20234227080@stu.suda.edu.cn organization: Soochow University,School of Computer Science and Technology,China – sequence: 2 givenname: Yanhui surname: Li fullname: Li, Yanhui email: yanhuili@nju.edu.cn organization: Nanjing University,State Key Laboratory for Novel Software Technology,China – sequence: 3 givenname: Jun surname: Wang fullname: Wang, Jun email: 602022330025@smail.nju.edu.cn organization: Nanjing University,State Key Laboratory for Novel Software Technology,China – sequence: 4 givenname: Xiaofang surname: Zhang fullname: Zhang, Xiaofang email: xfzhang@suda.edu.cn organization: Soochow University,School of Computer Science and Technology,China |
| BookMark | eNpVjr1OwzAURg0CiVIyszDkBVL8c23XI6pKQWrFQBiYqhvHaS0lThWHSnl7jGBhOp_OcPTdkqvQB0fIPaMLxkA-CmWY4nSRKKkyFyQz2iyBUs04LPUlmXEFomBS8xuSxegrmqZUjKkZ-Vyfsf3C0YdDXrqh86Fv-8OUlwOG2Cbfh9yHfIf26IP7p9-nOLou5meP-c6N2PXD6ehtysSf3B25brCNLvvjnHw8r8vVS7F927yunrYFpndjUVe8tpwzIx2AAMGbilIAJxBQGGDpqwVWUwtKYQOSN1TXQlUa7dJK5cScPPx2vXNufxp8h8O0Z1QrMFyIb7Y2VR4 |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IL CBEJK RIE RIL |
| DOI | 10.1145/3691620.3695069 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP All) 1998-Present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EISBN | 9798400712487 |
| EISSN | 2643-1572 |
| EndPage | 769 |
| ExternalDocumentID | 10764923 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: National Natural Science Foundation of China funderid: 10.13039/501100001809 |
| GroupedDBID | 6IE 6IF 6IH 6IK 6IL 6IM 6IN 6J9 AAJGR AAWTH ABLEC ACREN ADYOE ADZIZ AFYQB ALMA_UNASSIGNED_HOLDINGS AMTXH BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK IPLJI M43 OCL RIE RIL |
| ID | FETCH-LOGICAL-a248t-db2dc22195e443432fb0044e3a4a3941057c41d0c466af452f07d36b7ac8c56e3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 1 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001353105400061&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Jan 15 06:20:43 EST 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a248t-db2dc22195e443432fb0044e3a4a3941057c41d0c466af452f07d36b7ac8c56e3 |
| PageCount | 12 |
| ParticipantIDs | ieee_primary_10764923 |
| PublicationCentury | 2000 |
| PublicationDate | 2024-Oct.-27 |
| PublicationDateYYYYMMDD | 2024-10-27 |
| PublicationDate_xml | – month: 10 year: 2024 text: 2024-Oct.-27 day: 27 |
| PublicationDecade | 2020 |
| PublicationTitle | IEEE/ACM International Conference on Automated Software Engineering : [proceedings] |
| PublicationTitleAbbrev | ASE |
| PublicationYear | 2024 |
| Publisher | ACM |
| Publisher_xml | – name: ACM |
| SSID | ssib057256116 ssj0051577 |
| Score | 2.2793152 |
| Snippet | Machine translation has become an integral part of daily life, with terminology translation playing a crucial role in ensuring the accuracy of translation... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 758 |
| SubjectTerms | Accuracy Dictionaries Internet Limiting Machine translation Measurement Metamorphic Testing Software Software engineering Software Testing Terminology Translation Testing |
| Title | Evaluating Terminology Translation in Machine Translation Systems via Metamorphic Testing |
| URI | https://ieeexplore.ieee.org/document/10764923 |
| WOSCitedRecordID | wos001353105400061&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV09T8MwELWgYmAqH0V8ywNrIHFsXzyjVgy06lCkMlXO5YIykKI27e_HdlIqBga2yINl-XL3Ls57z4w9SNLo-lQdyQQwkgXaKMswjsoYCoW5cckZhMKvMJlk87mZdmL1oIUhokA-o0f_GP7lF0vc-KMyl-GgvaHYITsE0K1Ya_fyKHDgnfhepy3DDqcBOi-fRKqnVLtGSLhvVG1UHOjN-8tUApaM-v9cxQkb7FV5fPqDN6fsgOoz1t9dy8C7LD1n78POwbv-4LPAdQlH5zzAUkt941XNx4FGSb-GO_9yvq0sH1NjP5cuDBW6adZ-ugF7Gw1nzy9Rd4NCZIXMmqjIRYHCFSVFMkhIS5-lklIrbWo8wxNQJkWMUmtbSiV8iFKdg8UMlab0gvXqZU2XjJucRI6ihNL78YA1SGniqpOSUBpZ0hUb-K1afLUmGYvdLl3_MX7DjoXrDzwMCLhlvWa1oTt2hNumWq_uQ2i_AdIkpQk |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV09T8MwED1BQYKpfBTxjQfWQOKc7WRGRUW0VYcilalynAvqQIratL8f20mpGBjYIg-W5cvduzjvPQPcI0lj-1QZYKRMgLnRQZKYMChClQuTpTY5vVC4r4bDZDJJR41Y3WthiMiTz-jBPfp_-fncrNxRmc1wJZ2h2C7sCUQe1nKtzesjlIXvyHU7dSG2SK1U4-YToXiMpW2FuP1KlakIPcF5e52KR5Pn9j_XcQSdrS6PjX4Q5xh2qDyB9uZiBtbk6Sm8dxsP7_KDjT3bxR-eMw9MNfmNzUo28ERK-jXcOJiz9UyzAVX6c24DMTN2mqWbrgNvz93xUy9o7lAINMekCvKM54bbsiQIvYi0cHmKFGvUceo4nspglIcGpdQFCu6CFMtMaZMYISk-g1Y5L-kcWJoRzwwvVOEceZRODcWRrU8CVZFiQRfQcVs1_aptMqabXbr8Y_wODnrjQX_afxm-XsEht92CAwWurqFVLVZ0A_tmXc2Wi1sf5m-WKKhQ |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=IEEE%2FACM+International+Conference+on+Automated+Software+Engineering+%3A+%5Bproceedings%5D&rft.atitle=Evaluating+Terminology+Translation+in+Machine+Translation+Systems+via+Metamorphic+Testing&rft.au=Xu%2C+Yihui&rft.au=Li%2C+Yanhui&rft.au=Wang%2C+Jun&rft.au=Zhang%2C+Xiaofang&rft.date=2024-10-27&rft.pub=ACM&rft.eissn=2643-1572&rft.spage=758&rft.epage=769&rft_id=info:doi/10.1145%2F3691620.3695069&rft.externalDocID=10764923 |