Structure-Invariant Testing for Machine Translation
In recent years, machine translation software has increasingly been integrated into our daily lives. People routinely use machine translation for various applications, such as describing symptoms to a foreign doctor and reading political news in a foreign language. However, the complexity and intrac...
Uloženo v:
| Vydáno v: | 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE) s. 961 - 973 |
|---|---|
| Hlavní autoři: | , , |
| Médium: | Konferenční příspěvek |
| Jazyk: | angličtina |
| Vydáno: |
ACM
01.10.2020
|
| Témata: | |
| ISSN: | 1558-1225 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | In recent years, machine translation software has increasingly been integrated into our daily lives. People routinely use machine translation for various applications, such as describing symptoms to a foreign doctor and reading political news in a foreign language. However, the complexity and intractability of neural machine translation (NMT) models that power modern machine translation make the robustness of these systems difficult to even assess, much less guarantee. Machine translation systems can return inferior results that lead to misunderstanding, medical misdiagnoses, threats to personal safety, or political conflicts. Despite its apparent importance, validating the robustness of machine translation systems is very difficult and has, therefore, been much under-explored. To tackle this challenge, we introduce structure-invariant testing (SIT), a novel metamorphic testing approach for validating machine translation software. Our key insight is that the translation results of "similar" source sentences should typically exhibit similar sentence structures. Specifically, SIT (1) generates similar source sentences by substituting one word in a given sentence with semantically similar, syntactically equivalent words; (2) represents sentence structure by syntax parse trees (obtained via constituency or dependency parsing); (3) reports sentence pairs whose structures differ quantitatively by more than some threshold. To evaluate SIT, we use it to test Google Translate and Bing Microsoft Translator with 200 source sentences as input, which led to 64 and 70 buggy issues with 69.5% and 70% top-1 accuracy, respectively. The translation errors are diverse, including under-translation, over-translation, incorrect modification, word/phrase mistranslation, and unclear logic. |
|---|---|
| AbstractList | In recent years, machine translation software has increasingly been integrated into our daily lives. People routinely use machine translation for various applications, such as describing symptoms to a foreign doctor and reading political news in a foreign language. However, the complexity and intractability of neural machine translation (NMT) models that power modern machine translation make the robustness of these systems difficult to even assess, much less guarantee. Machine translation systems can return inferior results that lead to misunderstanding, medical misdiagnoses, threats to personal safety, or political conflicts. Despite its apparent importance, validating the robustness of machine translation systems is very difficult and has, therefore, been much under-explored. To tackle this challenge, we introduce structure-invariant testing (SIT), a novel metamorphic testing approach for validating machine translation software. Our key insight is that the translation results of "similar" source sentences should typically exhibit similar sentence structures. Specifically, SIT (1) generates similar source sentences by substituting one word in a given sentence with semantically similar, syntactically equivalent words; (2) represents sentence structure by syntax parse trees (obtained via constituency or dependency parsing); (3) reports sentence pairs whose structures differ quantitatively by more than some threshold. To evaluate SIT, we use it to test Google Translate and Bing Microsoft Translator with 200 source sentences as input, which led to 64 and 70 buggy issues with 69.5% and 70% top-1 accuracy, respectively. The translation errors are diverse, including under-translation, over-translation, incorrect modification, word/phrase mistranslation, and unclear logic. |
| Author | Su, Zhendong Meister, Clara He, Pinjia |
| Author_xml | – sequence: 1 givenname: Pinjia surname: He fullname: He, Pinjia email: pinjia.he@inf.ethz.ch organization: ETH Zurich,Department of Computer Science,Switzerland – sequence: 2 givenname: Clara surname: Meister fullname: Meister, Clara email: clara.meister@inf.ethz.ch organization: ETH Zurich,Department of Computer Science,Switzerland – sequence: 3 givenname: Zhendong surname: Su fullname: Su, Zhendong email: zhendong.su@inf.ethz.ch organization: ETH Zurich,Department of Computer Science,Switzerland |
| BookMark | eNotzD1PwzAQgGGDQKIpzAws-QMpPl9c2yOq-KhUxECYq8v1AkbFQY6LxL8HCaZXeoa3UidpTKLUJegFQGuvEZ3zAAtErxHDkap-VaMDA3isZmCtb8AYe6aqaXrXWi_bEGYKn0s-cDlkadbpi3KkVOpOphLTaz2MuX4kfotJ6i5TmvZU4pjO1elA-0ku_jtXL3e33eqh2Tzdr1c3m4ZM60pjCYIE551lY7XeYc8GAge_454do7E0MDL3IoDYWu9bohC82OVgWA84V1d_3ygi288cPyh_b4PxrdYGfwA3fkWs |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IH CBEJK RIE RIO |
| DOI | 10.1145/3377811.3380339 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Xplore: IEL IEEE Proceedings Order Plans (POP) 1998-present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE/IET Electronic Library url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EISBN | 1450371213 9781450371216 |
| EISSN | 1558-1225 |
| EndPage | 973 |
| ExternalDocumentID | 9284002 |
| Genre | orig-research |
| GroupedDBID | -~X .4S .DC 123 23M 29O 5VS 6IE 6IF 6IH 6IK 6IL 6IM 6IN 8US AAJGR AAWTH ABLEC ADZIZ AFFNX ALMA_UNASSIGNED_HOLDINGS APO ARCSS AVWKF BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO EDO FEDTE I-F I07 IEGSK IJVOP IPLJI M43 OCL RIE RIL RIO RNS XOL |
| ID | FETCH-LOGICAL-a247t-5a19e97875c2500d3bc219c98dcbc7c325afc3ccbee13345884aa998e56f2c0f3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 69 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000652529800079&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Aug 27 02:32:58 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a247t-5a19e97875c2500d3bc219c98dcbc7c325afc3ccbee13345884aa998e56f2c0f3 |
| PageCount | 13 |
| ParticipantIDs | ieee_primary_9284002 |
| PublicationCentury | 2000 |
| PublicationDate | 2020-Oct. |
| PublicationDateYYYYMMDD | 2020-10-01 |
| PublicationDate_xml | – month: 10 year: 2020 text: 2020-Oct. |
| PublicationDecade | 2020 |
| PublicationTitle | 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE) |
| PublicationTitleAbbrev | ICSE |
| PublicationYear | 2020 |
| Publisher | ACM |
| Publisher_xml | – name: ACM |
| SSID | ssj0006499 ssj0002870079 |
| Score | 2.4986947 |
| Snippet | In recent years, machine translation software has increasingly been integrated into our daily lives. People routinely use machine translation for various... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 961 |
| SubjectTerms | Machine translation Metamorphic testing Structural invariance |
| Title | Structure-Invariant Testing for Machine Translation |
| URI | https://ieeexplore.ieee.org/document/9284002 |
| WOSCitedRecordID | wos000652529800079&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV09T8MwELVKxcBUoEV8ywMjbhOfE8czooKBqhJF6lY5lytiSVFp-_t7dkMREgtLZGXyh3x3z3fvnRB3ZW4K0miUcRV_OGBVzmdOJQCuSj2hJozNJuxoVEynbtwS93suDBHF4jPqh2HM5VcLXIensoFjWxqVIw-szXdcrf17SkjYJSHj1FjhnEP5RsonNdkAwAZOZZ8RWZjGr14q0ZUMO_-bxLHo_XDy5HjvbU5Ei-pT0fluyiCbO9oV8BoVYddLUs_1hpEwb52cBC2N-l1yhCpfYvkkyeildpVwPfE2fJw8PKmmM4Ly2tiVynzqiPGfzZBDmKSCEtnyoCsqLNEi6MzPERBLIsagkYzqPQMryvK5xmQOZ6JdL2o6F9J6QGN1GTIuxqdQFoBgEnJQYOmNvRDdsAezz534xaxZ_uXfv6_EkQ6ANFa7XYs2r5huxCFuVh9fy9t4Ylvl-JWE |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NTwIxEJ0QNNETKhi_7cGjhd22S7dnI4EIhERMuJF2djBeVoPA77ctK8bEi5dNs6d-pDPzOjPvAdy5rspJoOLKFP7jA1ZubGZ4IqUpUksoCKPYhB6P89nMTGpwv-uFIaJYfEbtMIy5_OId1-GprGO8LY3MkXtBOavq1tq9qISUXRJyTpUd7vpgviLzSVXWkVKHrsq2x2RhIr_UVKIz6TX-N40jaP105bHJzt8cQ43KE2h8yzKw6pY2QT5HTtj1kvig3Hgs7DePTQObRvnKfIzKRrGAklj0U9tauBa89B6nD31eaSNwK5Re8cymhjwC1Bn6ICYppENve9DkBTrUKEVmFygRHZFHobEd1VoPrSjrLgQmC3kK9fK9pDNg2kpUWriQc1E2lS6XKFVCRuborNLn0Ax7MP_Y0l_Mq-Vf_P37Fg7609FwPhyMny7hUAR4GmvfrqDuV0_XsI-b1dvn8iae3hfkD5jN |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=proceeding&rft.title=2020+IEEE%2FACM+42nd+International+Conference+on+Software+Engineering+%28ICSE%29&rft.atitle=Structure-Invariant+Testing+for+Machine+Translation&rft.au=He%2C+Pinjia&rft.au=Meister%2C+Clara&rft.au=Su%2C+Zhendong&rft.date=2020-10-01&rft.pub=ACM&rft.eissn=1558-1225&rft.spage=961&rft.epage=973&rft_id=info:doi/10.1145%2F3377811.3380339&rft.externalDocID=9284002 |