Transliteration normalization for Information Extraction and Machine Translation

Foreign name transliterations typically include multiple spelling variants. These variants cause data sparseness and inconsistency problems, increase the Out-of-Vocabulary (OOV) rate, and present challenges for Machine Translation, Information Extraction and other natural language processing (NLP) t...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Journal of King Saud University. Computer and information sciences Ročník 26; číslo 4; s. 379 - 387
Hlavní autoři: Yuval Marton, Imed Zitouni
Médium: Journal Article
Jazyk:angličtina
Vydáno: Springer 01.12.2014
Témata:
Arabic
Information Extraction
Machine Translation
Name normalization
Named Entity Recognition
Transliteration
ISSN:1319-1578
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Foreign name transliterations typically include multiple spelling variants. These variants cause data sparseness and inconsistency problems, increase the Out-of-Vocabulary (OOV) rate, and present challenges for Machine Translation, Information Extraction and other natural language processing (NLP) tasks. This work aims to identify and cluster name spelling variants using a Statistical Machine Translation method: word alignment. The variants are identified by being aligned to the same “pivot” name in another language (the source-language in Machine Translation settings). Based on word-to-word translation and transliteration probabilities, as well as the string edit distance metric, names with similar spellings in the target language are clustered and then normalized to a canonical form. With this approach, tens of thousands of high-precision name transliteration spelling variants are extracted from sentence-aligned bilingual corpora in Arabic and English (in both languages). When these normalized name spelling variants are applied to Information Extraction tasks, improvements over strong baseline systems are observed. When applied to Machine Translation tasks, a large improvement potential is shown.
ISSN:1319-1578
DOI:10.1016/j.jksuci.2014.06.011