The structure of unseen trigrams and its application to language models: A first investigation

In a series of preparatory experiments in 4 languages on subsets of the Europarl corpus, we show that a large number of unseen trigrams can be reconstructed by proportional analogy with trigrams having the lowest frequencies. We derive a very simple smoothing scheme from this empirical result and sh...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:2010 4th International Universal Communication Symposium s. 273 - 280
Hlavní autoři: Lepage, Y, Gosme, J, Lardilleux, A
Médium: Konferenční příspěvek
Jazyk:angličtina
japonština
Vydáno: IEEE 01.10.2010
Témata:
Connectors
Electronic mail
Estimation
Europarl
Morphology
Smoothing methods
structure of unseen trigrams
Training
Training data
Trigram language models
ISBN:9781424478217, 1424478219
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:In a series of preparatory experiments in 4 languages on subsets of the Europarl corpus, we show that a large number of unseen trigrams can be reconstructed by proportional analogy with trigrams having the lowest frequencies. We derive a very simple smoothing scheme from this empirical result and show that it outperforms Good-Turing and Kneser-Ney smoothing schemes on trigrams models in all 11 languages on the common multilingual part of the Europarl corpus, except Finnish.
ISBN:9781424478217
1424478219
DOI:10.1109/IUCS.2010.5666011