Maximum Likelihood Estimation of Functionals of Discrete Distributions

We consider the problem of estimating functionals of discrete distributions, and focus on a tight (up to universal multiplicative constants for each specific functional) nonasymptotic analysis of the worst case squared error risk of widely used estimators. We apply concentration inequalities to anal...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:IEEE transactions on information theory Ročník 63; číslo 10; s. 6774 - 6798
Hlavní autori: Jiantao Jiao, Venkat, Kartik, Yanjun Han, Weissman, Tsachy
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: New York IEEE 01.10.2017
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Predmet:
Approximation methods
approximation theory
approximation using positive linear operators
Bayesian analysis
Complexity theory
Constants
Dirichlet prior smoothing
Dirichlet problem
Economic models
Entropy
Entropy (Information theory)
Entropy estimation
Estimating techniques
Estimation
high dimensional statistics
Information theory
Linear operators
Mathematical analysis
Maximum likelihood estimation
maximum likelihood estimator
Maximum likelihood estimators
Minimax technique
Risk
Rényi entropy
Smoothing methods
Variations
ISSN:0018-9448, 1557-9654
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:We consider the problem of estimating functionals of discrete distributions, and focus on a tight (up to universal multiplicative constants for each specific functional) nonasymptotic analysis of the worst case squared error risk of widely used estimators. We apply concentration inequalities to analyze the random fluctuation of these estimators around their expectations and the theory of approximation using positive linear operators to analyze the deviation of their expectations from the true functional, namely their bias. We explicitly characterize the worst case squared error risk incurred by the maximum likelihood estimator (MLE) in estimating the Shannon entropy H(P) = Σ i=1 S -p i ln p i , and the power sum F α (P) = Σ i=1 S p i α , α > 0, up to universal multiplicative constants for each fixed functional, for any alphabet size S ≤ ∞ and sample size n for which the risk may vanish. As a corollary, for Shannon entropy estimation, we show that it is necessary and sufficient to have n ≫ S observations for the MLE to be consistent. In addition, we establish that it is necessary and sufficient to consider n ≫ S 1/α samples for the MLE to consistently estimate F α (P), 0 <; α <; 1. The minimax rate-optimal estimators for both problems require S/ ln S and S 1/α / ln S samples, which implies that the MLE has a strictly sub-optimal sample complexity. When 1 <; α <; 3/2, we show that the worst case squared error rate of convergence for the MLE is n -2(α-1) for infinite alphabet size, while the minimax squared error rate is (n ln n) -2(α-1) . When α ≥ 3/2, the MLE achieves the minimax optimal rate n -1 regardless of the alphabet size. As an application of the general theory, we analyze the Dirichlet prior smoothing techniques for Shannon entropy estimation. In this context, one approach is to plug-in the Dirichlet prior smoothed distribution into the entropy functional, while the other one is to calculate the Bayes estimator for entropy under the Dirichlet prior for squared error, which is the conditional expectation. We show that in general such estimators do not improve over the maximum likelihood estimator. No matter how we tune the parameters in the Dirichlet prior, this approach cannot achieve the minimax rates in entropy estimation. The performance of the minimax rate-optimal estimator with n samples is essentially at least as good as that of Dirichlet smoothed entropy estimators with n ln n samples.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0018-9448
1557-9654
DOI:10.1109/TIT.2017.2733537