The structure of unseen trigrams and its application to language models: A first investigation

In a series of preparatory experiments in 4 languages on subsets of the Europarl corpus, we show that a large number of unseen trigrams can be reconstructed by proportional analogy with trigrams having the lowest frequencies. We derive a very simple smoothing scheme from this empirical result and sh...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:2010 4th International Universal Communication Symposium s. 273 - 280
Hlavní autori: Lepage, Y, Gosme, J, Lardilleux, A
Médium: Konferenčný príspevok..
Jazyk:English
Japanese
Vydavateľské údaje: IEEE 01.10.2010
Predmet:
Connectors
Electronic mail
Estimation
Europarl
Morphology
Smoothing methods
structure of unseen trigrams
Training
Training data
Trigram language models
ISBN:9781424478217, 1424478219
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:In a series of preparatory experiments in 4 languages on subsets of the Europarl corpus, we show that a large number of unseen trigrams can be reconstructed by proportional analogy with trigrams having the lowest frequencies. We derive a very simple smoothing scheme from this empirical result and show that it outperforms Good-Turing and Kneser-Ney smoothing schemes on trigrams models in all 11 languages on the common multilingual part of the Europarl corpus, except Finnish.
ISBN:9781424478217
1424478219
DOI:10.1109/IUCS.2010.5666011