The structure of unseen trigrams and its application to language models: A first investigation

In a series of preparatory experiments in 4 languages on subsets of the Europarl corpus, we show that a large number of unseen trigrams can be reconstructed by proportional analogy with trigrams having the lowest frequencies. We derive a very simple smoothing scheme from this empirical result and sh...

Full description

Saved in:
Bibliographic Details
Published in:2010 4th International Universal Communication Symposium pp. 273 - 280
Main Authors: Lepage, Y, Gosme, J, Lardilleux, A
Format: Conference Proceeding
Language:English
Japanese
Published: IEEE 01.10.2010
Subjects:
Connectors
Electronic mail
Estimation
Europarl
Morphology
Smoothing methods
structure of unseen trigrams
Training
Training data
Trigram language models
ISBN:9781424478217, 1424478219
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In a series of preparatory experiments in 4 languages on subsets of the Europarl corpus, we show that a large number of unseen trigrams can be reconstructed by proportional analogy with trigrams having the lowest frequencies. We derive a very simple smoothing scheme from this empirical result and show that it outperforms Good-Turing and Kneser-Ney smoothing schemes on trigrams models in all 11 languages on the common multilingual part of the Europarl corpus, except Finnish.
ISBN:9781424478217
1424478219
DOI:10.1109/IUCS.2010.5666011