The structure of unseen trigrams and its application to language models: A first investigation

In a series of preparatory experiments in 4 languages on subsets of the Europarl corpus, we show that a large number of unseen trigrams can be reconstructed by proportional analogy with trigrams having the lowest frequencies. We derive a very simple smoothing scheme from this empirical result and sh...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:2010 4th International Universal Communication Symposium S. 273 - 280
Hauptverfasser: Lepage, Y, Gosme, J, Lardilleux, A
Format: Tagungsbericht
Sprache:Englisch
Japanisch
Veröffentlicht: IEEE 01.10.2010
Schlagworte:
Connectors
Electronic mail
Estimation
Europarl
Morphology
Smoothing methods
structure of unseen trigrams
Training
Training data
Trigram language models
ISBN:9781424478217, 1424478219
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:In a series of preparatory experiments in 4 languages on subsets of the Europarl corpus, we show that a large number of unseen trigrams can be reconstructed by proportional analogy with trigrams having the lowest frequencies. We derive a very simple smoothing scheme from this empirical result and show that it outperforms Good-Turing and Kneser-Ney smoothing schemes on trigrams models in all 11 languages on the common multilingual part of the Europarl corpus, except Finnish.
ISBN:9781424478217
1424478219
DOI:10.1109/IUCS.2010.5666011